Kafka - Ingestion and Processing Pipeline

Agenda

Kafka and Cloudera - Shravan Pabba

• Concepts and Architecture

• Kafka vs Traditional messaging systems

• Install and configure Kafka on Cloudera cluster

• Client tools - Add and consume data from topics

• Replication and Failover capabilities (Demo)

• Monitoring Kafka using Cloudera Manager

• Flume Integration and demo of Kafka to Flume to HDFS

• API's, Mirror Maker, Camus, Samza