Apache Kafka: Optimizing Your Deployment

Session Abstract:

Understanding and adhering to the latest best practices for leveraging the powerful open source Apache Kafka data streaming platform will ensure that your deployment is far more effective, efficient, and simple to manage.

For example, Kafka’s low overhead and horizontal-scaling-friendly design enables the solution to run successfully even on inexpensive commodity hardware – a significant advantage that developers should be well aware of when preparing their deployments. At the same time, utilizing Apache ZooKeeper and providing it with ample bandwidth and powerful disks (while storing logs storing logs separately, isolating the ZooKeeper process, and disabling swaps), will serve to minimize latency. Likewise, increasing Kafka’s default replication factor from two to three, using partitions correctly, and monitoring system metrics will result in far smoother operations.

This session will provide developers with specific and very actionable steps for optimizing their Kafka deployments and realizing more of the open source technology’s potential. Attendees already using or considering open source Apache Kafka will come away with acute knowledge of how to:

• Set log configuration parameters to keep logs manageable
• Take advantage of Kafka’s (low) hardware requirements
• Leverage Apache ZooKeeper to its fullest
• Set up replication and redundancy the right way
• Be careful with topic configurations
• Use parallel processing
• Configure and isolate Kafka with security in mind
• Avoid outages by raising the Ulimit
• Maintain a low network latency
• Utilize effective monitoring and alerts

Speaker Bio for Zeke Dean, Senior Consultant at Instaclustr
Zeke Dean is a Senior Consultant at Instaclustr. He has numerous years of experience building large big data systems for major global enterprises, from banks in the Middle East to major publishing houses in the US. Zeke works with Instaclustr clients all over the world to solve their messaging, streaming, and integration requirements using Kafka.

Prior to joining Instaclusr, Zeke worked at MAVRCK where he did social media profiling using Kafka workflows to derive values for influencer marketing. His most recent role was in the Middle East architecting large financial reconciliation systems and a microlending platform for the burgeoning middle-class in India.

Co-sponsored with Instaclustr - food and drinks provided