addressalign-toparrow-leftarrow-rightbackbellblockcalendarcameraccwcheckchevron-downchevron-leftchevron-rightchevron-small-downchevron-small-leftchevron-small-rightchevron-small-upchevron-upcircle-with-checkcircle-with-crosscircle-with-pluscontroller-playcrossdots-three-verticaleditemptyheartexporteye-with-lineeyefacebookfolderfullheartglobegmailgooglegroupshelp-with-circleimageimagesinstagramFill 1light-bulblinklocation-pinm-swarmSearchmailmessagesminusmoremuplabelShape 3 + Rectangle 1ShapeoutlookpersonJoin Group on CardStartprice-ribbonprintShapeShapeShapeShapeImported LayersImported LayersImported Layersshieldstartickettrashtriangle-downtriangle-uptwitteruserwarningyahoo

Devops Singapore Meetup #16: Managing 1000 instances on EC2 with Capistrano

Presenter: Vinoth Gopi, Semantics3

The session:
Semantics3 is currently running close to 1000 machines on EC2 and processing over 10TB of data a day. They have rolled out their own distributed computing infrastructure that crawls, normalizes and disambiguates data in a continuous pipeline - analagous to Hadoop, but not exactly the same. All this, managed by just 5 people. Based on demand, the engineers can spin up machines in clusters within a few minutes. All this is possible by using a little known feature of capistrano (the deployment tool) called host blocks. This combined with the EC2 API SDK for ruby, devops people can come up with their own deployment and auto-scaling solutions.

The talk will cover:
- A quick overview of our distributed framework, DBs and technologies used to build the backend
- How we actually manage the deployment and spin down of worker instances (semi-automatically) using capistrano and aws-sdk

More about the speaker:
Vinoth is one of the co-founder of Semantics3. His company has set out to achieve the audacious goal of organizing the world's e-commerce data by tracking all products sold online and their prices. They have now built a complex system that tracks over 35 million products sold across 22,000 merchants in the US and UK. Their customers range from small developers and startups building e-commerce applications to Fortune 500 companies who use their data to gain competitive intelligence. In January 2013 they were accepted into the Y Combinator program in the US, making them the first Singapore-based startup to join the prestigious startup program. They now have offices in San Francisco and Singapore.

Join or login to comment.

Our Sponsors

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy