Mar 29, 2012 · 7:00 PM
STAT Search Analytics is a Vancouver-based SEO Analytics company that powers real-time search data and analytics for SEO experts at some of the world's largest companies. Using Ruby, we crawl over 3 million keywords every day. We analyze that data in real time, and present it to our users via our application powered by Rails and extJS. For this talk, we'll cover the following:
- The Crawl
- How do we scale ruby to crawl 3mm keywords per day?
- What are some of the Ruby-related pitfalls we encountered? What lessons did we learn?
- The Storage
- We're analyzing over 100 GB of data per day. How do we store it all? Where do we store it?
- How do we prepare that data for display in Rails?
- How do we analyze all that data using Ruby?
- The Frontend
- How we built our own caching system to reduce Rails' load on our databases.
Born and raised in Vancouver, Rob Bucci is the Founder and CEO of STAT Search Analytics (getSTAT.com). Since 2008, STAT has equipped SEO experts with massive-scale, real-time, localised search data. Largely through word of mouth, the company has been happily building up a client base that includes some of the world's largest online companies and SEO agencies.
Rob’s been working with Ruby on Rails since 2006, and he’s been lucky enough to use it in a number of big-data projects related to SEO, data mining, and natural language processing.