Data Ingest and Processing - spotlight on Streaming

Data Ingest and Processing:

A lot of companies are looking to reduce the time it takes to get from ingest to intelligence with their critical business data. The challenges around moving from batch-focused processing to realtime/micro-batch can be difficult for both startups and established organizations. With these sessions, we will try to explore some solutions that have come up at various companies.

Presentations:

1. Stateful Stream Processing with Kafka and Samza (50min)

Intergration with in-memory local state is one of Samza's most interesting features, but how do you maintain and update the local state with fault-tolerance and multi-tenancy in mind ? How do you test it? We will talk about our solutions and problems to be solved

Speaker Bio:

George Li

Software Team Lead @ Vericent, an IBM Company

• a little Schemer, diehard fan of "How to Solve It" by George Pólya

2. Moving to a Realtime Ingestion and Processing Architecture (50min)

Postponed until next free presentation meetup.

3. Realtime Streaming Analytics

Topic: Real-time Streaming AnalyticsIf we look at where time to business insights from data is being significantly delayed in the entire analytics modeling life cycle, we can easily identify several areas. This presentation identify model deployment and execution as the two major bottlenecks and how it can be solved using a standards-based approach. It will cover both batch, real-time and streaming analytics.

Eddie Soong

A software engineer by training, transitioned to business development for enterprise software and self taught big data analytics enthusiast. Working experience in Data Management to BI to big data predictive analytics. A member of Zementis, a standards-based predictive model deployment and execution engine on big data infrastructure for batch and real-time scoring

Note: We are going to move our Q&A session to the Elephant and Caste so that thirsty THUGs can converse and so our presentations finish on time. :)

Please excuse the summer hiatus, we are back on track. :)