Kafka November Meetup

Streaming link: http://www.ustream.tv/linkedin-events

6:00 - Networking

6:30 - Opening - Clark Haskins (LinkedIn)

6:35 - Apache Kafka Security - Harsha Chintalapani & Parth Brahmbhatt (HortonWorks)

The number one feature request for Apache Kafka is security. As Apache Kafka is being adopted by more and more enterprises and IoT solutions, security feature has become critical feature for privacy and trust. In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. New security mechanism is compatible with rest of Hadoop security .

In this session we will cover the goals, internal design of security features and the best practices for using this in production.​

7:10 - Quotas - Aditya Auradkar (LinkedIn)

Starting in 0.9, Apache Kafka will introduce support for application quotas. We will discuss the the motivation, design, implementation and share some experimental results on traffic at LinkedIn.

7:35 - Take-aways from a live site issue - Joel Koshy (LinkedIn)

LinkedIn’s Kafka deployment is nearing 1300 brokers that move close to 1.3 trillion messages (https://engineering.linkedin.com/apache-kafka/how-we_re-improving-and-advancing-kafka-linkedin) a day. While operating Kafka smoothly even at this scale is testament to both Kafka’s scalability and the operational expertise of LinkedIn SREs we occasionally run into some very interesting bugs at this scale. In this talk I will dive into a production issue that we recently encountered as an example of how even a subtle bug can suddenly manifest at scale and cause a near meltdown of the cluster. We will go over how we detected and responded to the situation, investigated it after the fact and summarize some lessons learned and best-practices from this incident.

8:00 - Q&A