Every day, Mozilla collects three million Firefox crash reports from around the world. The data in these reports drives the bug-fixing priorities of Firefox engineers, and is critical to understanding the stability of our platform. In this case study, I'll describe the challenges we've faced, the types of questions the system can be used to answer, and the architecture and infrastructure we use to process, store, and analyze approximately 110TB of crash reports using Python, Hadoop, PostgreSQL, and a few other things thrown in for good measure.

All the software we use in our stack is Open Source, including the Breakpad client embedded in the browser, and the Socorro collection and reporting system. Other projects and companies are now using the Breakpad/Socorro combination.

Laura Thomson is an Engineering Manager at Mozilla. She is the co-author of "PHP and MySQL Web Development" and "MySQL Tutorial". She is a veteran speaker at Open Source conferences world wide.