How we Computer: Adventures in Cloud Infrastructure at Librato

The growth of ephemeral infrastructure has had an astounding impact on IT; from slashing the cost of product development to obliterating traditional notions about the lines between engineering disciplines.  Today, distributed systems architecture is composed of a dizzying array of components like message queues and stream processing systems that were unheard of in contemporary infrastructure design 5 years ago. In this talk we'll examine Librato's system architecture, which is entirely implemented on AWS and relies heavily on Apache’s Storm and Cassandra projects to sustain several hundred thousand http POST operations per second and store around 10 billion data points. We give infrastructure details, recounting our initial design decisions as well as various scaling challenges that have forced us to refactor our storage designs. Finally we relate the system and performance metrics that we’ve found useful in monitoring our storage infrastructure.

About Dave Josephsen

As the developer evangelist for Librato, Dave Josephsen hacks on tools and documentation, writes about statistics, systems monitoring, alerting, metrics collection and visualization, and generally does anything he can to help engineers and developers close the feedback loop in their systems. He's written books for Prentice Hall and O'Reilly, speaks shell, Go, C, Python, Perl and a little bit of Spanish (in that order), and has never lost a game of Calvinball.

Join or login to comment.

  • Yvonne

    Ahhh! I missed this. Was it video taped or are there any slides?

    July 16

    • Dave J.

      that's actually a pretty great Idea emad. I'll try to set aside some time today to just run through these with a mic and a screen-capture app, and get them up on youtube or vimeo.

      July 18

    • Dave J.

      So.. voice-over work is unexpectedly difficult. I've been trying to do this for like 5 hours and I basically hate the sound of my own voice now. Seriously, I may never speak again. Anyway, I said I would, so I'm still working on this and will get back to you with it soon. PS: If anyone happens to know Morgan Freeman's agent's number SMS me with it plsthnx

      July 23

  • Alan R.

    Dave - what would be some good cloud/AWS application architecture information sources you'd reco?

    July 17

    • Dave J.

      bleh, every time I hit return it closes the comment.. oh well. If you're aren't aware of the Legendary Gene Kim, then you should definitely purchase "the phoenix project" from amazon, and check him out on youtube. A good 20-minute overview is this one: https://www.youtube.co...­

      July 18

    • Dave J.

      doh. I hit return again, this is becoming a pavlovian experiment.. lets see how many more times I do it. Another fantastic blog is Kyle Kingsbery's series called "Jepson", http://aphyr.com/tags...­ I think somebody asked me a question to which I replied "all distributed systems problems are storage problems" and jepson is a fantastic resource for understanding why. I also personally follow the youtube and vimeo streams of several tech conferences including velocity, monitorama, and Omni-TI surge, and anything that USENIX puts out. I'll personally be speaking this year about myriad aspects of Librato's computational prowess at Velocity New York, DevopsDays Boston, Nagios World, and Usenix LISA, and my engineering team will have a booth at AWS: Reinvent this year, so if you're planning on going to any of those shoot a tweet at me and we'll grab a beer (or coffee or whatever).

      July 18

  • Sam W.

    Really good presentation this time. Very useful. Top notch.

    July 17

Our Sponsors

People in this
Meetup are also in:

Create your own Meetup Group

Get started Learn more
Rafaël

We just grab a coffee and speak French. Some people have been coming every week for months... it creates a kind of warmth to the group.

Rafaël, started French Conversation Group

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy