Handling hundreds of millions of search queries per day against a data set that normally lives in MySQL and is constantly changing and growing presents a number of interesting challenges. Things get even more fun when you'd like to move from "batch updates" of your search indexes to real-time. That's what we've done at craigslist in the last year using the open source Sphinx search engine along with MySQL and some custom Perl code to tie it all together.
This presentation will include:
• a brief overview of the sphinx search engine and why we use it
• a snapshot of our previous search architecture(s)
• our new search architecture
• lessons learned and pain points along the way
• new sphinx features that make real-time indexing more flexible
• ideas for future enhancements
Jeremy Zawodny (@jzawodn / blog) hacks on Sphinx, MySQL, Redis, MongoDB, Perl, and various back-end bits of the craigslist infrastructure. He flies both small airplanes and gliders in his spare time.
6:30 - 7:00 -- Doors open/general socializing & food provided by craigslist
7:00 - 7:15 -- Intro/announcements
7:15 - 8:15(ish) -- Main presentation
8:15 - 8:45 -- General discussion/Q&A/networking/etc.
Percona has made MySQL faster and more reliable for over 2,000 consulting and support customers worldwide since 2006. Percona provides enterprise-grade MySQL Support, Consulting, Training, Remote DBA, and Server Development services to companies including Cisco Systems, Alcatel-Lucent, Groupon and the BBC. Percona's founders authored the definitive book “High Performance MySQL” from O'Reilly Press and the widely read MySQL Performance Blog. Percona also develops software for MySQL users, including Percona Server, Percona XtraBackup, Percona XtraDB Cluster, and Percona Toolkit. The popular Percona Live conferences draw attendees and acclaimed speakers from around the world. For more information, visit www.percona.com
craigslist is a place to buy, sell, rent, hire, share, meet, swap, discuss, find, serve, connect, give away, announce, work, collect, care, perform, learn, marvel, mentor, befriend, fall in love and/or save the world.
Lithium helps companies unlock the passion of their customers. Lithium software powers amazing Social Customer Experiences for more than 400 iconic brands including AT&T, BT, Best Buy, Indosat, Sephora, and Telstra. Lithium helps companies grow brand advocacy, drive sales, reduce costs and accelerate innovation to create social communities that redefine the customer experience.
O'Reilly Media spreads the knowledge of innovators through its books, online services, magazines, research, and conferences. Since 1978, O'Reilly has been a chronicler and catalyst of leading-edge development, homing in on the technology trends that really matter and galvanizing their adoption by amplifying "faint signals" from the alpha geeks who are creating the future. An active participant in the technology community, the company has a long history of advocacy, meme-making, and evangelism.