Confluent Kafka Python: Integrating Python Apps with your Data Pipeline
Data is the new oil and is a vital component to the success of every modern business. That data must be highly accessible without impacting your core services. With the advent of Apache Kafka and the ability to write messages at high throughput and consume messages in parallel, it is easier then ever to get your data to the right stakeholders.
In this session, I'll provide an introduction to the Apache Kafka protocol and explain how messages are written and read from Apache Kafka. We'll then briefly discuss the message delivery guarantees and how fault tolerance plays a critical role.
We'll then dive into Confluent Kafka Python library, an open source Kafka client for Python applications. We'll first give an introduction to the APIs and then dig into some the best practices as you integrate and deploy it with your services. We will also cover some pitfalls and gotchas to avoid.
Presenter Bio: Mike Trienis loves building data products that scale. That means implementing simple solutions with minimal maintenance through automation and eloquent designs. My software experience spans the full stack; from system level deployment to application implementation. In particular, I have spent quite a bit of time working with streaming technologies such as Apache Kafka and Apache Spark.
