Simplify Your Architecture: Say No to Lambda, presented by VoltDB

http://photos1.meetupstatic.com/photos/event/d/5/9/8/600_436374680.jpeg

What is Lambda? The architecture is designed to handle massive quantities of data by taking advantage of both batch- and stream-processing methods. It proposes that both speed/streaming and batch workloads be run in parallel on the same incoming data. The speed layer can achieve faster responses, while the batch layer can be more robust and serve as the system of record for historical data. Lambda also requires a serving layer to serve results.

What’s a real-world example?

Ed Solovey of Twitter (formerly Crashlytics) has given several talks on the use of the Lambda Architecture for the Crashlytics service, including a 20-minute presentation given at the October Boston Facebook @Scale Conference ( http://youtu.be/56wy_mGEnzQ ).

The company needed to identify how many times end-users access a mobile app, which means handling hundreds of thousands of unique ids per second. To solve this problem the company implemented the Lambda Architecture. In the speed layer, they enlisted Kafka for ingestion, Storm for processing, Cassandra for state and Zookeeper for distributed agreement. In the batch layer, tuples were loaded in batches into S3, then processed with Cascading and Amazon Elastic Map-Reduce.

The problem? It’s complicated.

Sure, this approach can meet high-performance needs, but it comes with tremendous cost. Focusing on just the speed layer, getting Zookeeper, Kafka, Storm and Cassandra combined into a reliable fast data engine is expensive in developer time, computing resources and in operational ongoing support. Each system requires at least three nodes to run, meaning your speed layer is at least 12 nodes, and often larger. And once the speed layer is working, the batch layer is a second problem with its own complexity. Even with reliable components, the odds that any single component will have issues, goes up as the number of components rises. And when a component fails, how well trained is the operational support when there is such breadth to the app? Isolating which component is the issue is difficult when you must also consider inter-component interaction when hunting for problems.

What’s a better answer?

Simplify. Simplify. Simplify. Removing just a single component of a typical Rube-Goldbergian Lambda implementation can reduce complexity and cost, but also will make it easier to change the application as business needs change. Look to replace all or part of your Lambda stack with a more integrated solution and this discussion will show you how with a clear example. See how thousands of lines of code becomes 30 when you collapse disparate systems with those that integrate ingestion, state, agreement and processing.

About the Presenter

http://photos3.meetupstatic.com/photos/event/7/f/6/8/600_436352616.jpeg

Ryan Betts, CTO, VoltDB

Ryan was one of the initial developers of VoltDB’s commercial product, and values his unique opportunity to closely collaborate with customers, partners and prospects to understand their data management needs and help them to realize the business value of VoltDB and related technologies.

About the location:
Franz Hall - Room 026
Parking available in the main parking lot (no permit required)

Campus Map: https://pilots.up.edu/documents/1960960/2608978/Campus+Map.pdf