Apache Kafka at Ebay, Salesforce and Confluent

We are glad to announce that we have added a talk from Confluent for this meetup!

There are three great talks in this meetup. Not only will you learn about several business use cases solved using Kafka in companies like Salesforce and Ebay, but you will also learn about new popular features introduced in Kafka recently – Kafka Connect and Kafka Streams.

Ed Dudenhoefer from Ebay will talk about how Ebay uses Kafka in CMS Traffic delivery systems, which delivers messages/feeds from site-triggered streams and marketing bursts to email, mobile and other channels. If you ever receive any auction emails, marketing campaigns or notifications from Ebay, then they were probably rendered and delivered by CMS traffic delivery system.

Nishant from Salesforce will show us how they used Kafka in Salesforce to generate system health insights from host level metrics events.

Come and learn the new features introduced in Kafka – Kafka Connect and Kafka streams. Jeremy from Confluent will talk about how Kafka connect works and how can you write a stream processing pipeline using Kafka streams.

NOTE: If you are driving, please allow extra time for traffic, as there is construction activity near the building. We have a limited number of free parking validations for those parking in the building. (Entry to garage from 108th Ave NE).

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hope everyone is enjoying the great start to the summer. We have the next meetup session coming soon, with a couple of exciting talks from Salesforce and Ebay.

Please RSVP to confirm your attendance.

Agenda

5:30 pm Doors Open

5:30 to 6:00 Check-in, Food+Drinks, Networking

6:00 to 7:45 Talks

7:45 to 8:00 Networking & Wrap-Up

Session Details

Enabling real time scenarios at scale using Kafka at Salesforce

The Salesforce platform hosts web services and applications written by thousands of internal developers and millions of external developers to provide the largest business automation cloud on the planet. We process billions of customer interactions per day across our global data centers. Needless to say, operations at this scale generate petabytes of data.

The Salesforce DVA team—Diagnostics, Visibility and Analytics—builds the next generation web-scale, massively distributed platform for collecting, processing, transforming and analyzing this volume of data and providing near real-time operational and business insights to hundreds and thousands of delighted customers.

Kafka sits at the heart of the system architecture. Running across multiple global data centers, it processes hundreds of billions of events per day. In this talk, we cover the scenarios that it enables, architecture of the system, operational challenges and learnings, tools we use and where we are headed in the next year.

Speaker: Nishant Gupta

Nishant Gupta is a Sr. Director for Infrastructure Engineering at Salesforce. He is leading the team responsible for providing Kafka based data broker service. He brings expertise and experience in building and running mission-critical large scale distributed systems. Prior to joining Salesforce, Nishant held several positions in Engineering and Product Management at Microsoft.

Kafka Usage in Ebay Communications Delivery Pipeline

At Ebay, we use Kafka to loosely couple our event reception and processing machine pools. I will outline our Kafka cluster and topic configuration, and how we use custom partitioning and thread pools to achieve throughput for delivering 100+ million communications per day. Kafka is a central component in our communications delivery pipeline that processes 100 million messages per day, and will enable us to horizontally scale to 300+ million this year.

Speaker: Ed Dudenhoefer

Senior Software Engineer, Traffic Messaging Platform Team @ eBay, Inc.

I started my engineering life way back in the days of the Z-80, CP/M and when the Commodore-64 was “cutting edge”. I lived and worked in West Palm Beach on Pratt & Whitney’s YF-22 Raptor jet engine bid, Ann Arbor on the ArborText publishing system, and Providence on the EBT DynaText series when SGML was still cool and XML was still being formed by the W3C. It is amazing to see the search capabilities of the average desktop of today so easily outstrip indexing technology the best minds from Brown worked years at EBT to productize!

When I moved to Redmond in 1997 to work at Microsoft, I started on a team tasked with updating Windows in a secure way. This became Windows Update. I remember when “COM 2.0” and “Lightning” were code names for .NET in development, and was later part of Visual Studio when “Cool” was unveiled as C#. And for many years I was in SQL Server creating client tools such as Management Studio, and prototyping the earliest forays into SQL Azure.

I joined Amazon AWS in 2012 working on the Elastic Block Storage team – they certainly had the lead in cloud technology – but the commute from my home in Kirkland was driving me crazy. So I promised myself I would not work across that lake.

Luckily, Ebay has a beautiful office right in Bellevue. So I ride my Vespa to work, play club soccer, roller blade, and make circuit training classes at the Pro Club when I can. A stretch goal is to audition for and make it onto American Ninja Warrior. Life is good!

Kafka Connect and Kafka Streams

In this session I am introducing the audience to Kafka Connect and Kafka Streams. Kafka Connect is a Kafka's new tool for realtime, scalable, and fault-tolerant data import and export. Connect greatly simplifies importing data into Kafka topic which can be later consumed by downstream systems. Kafka Streams is a new lightweight and fault tolerant streaming library which integrates perfectly with reactive microservices.

Speaker: Jeremy Custenborder

Jeremy Custenborder is a Systems Engineer for Confluent, the company driving commercial Kafka support. He has consulted with big data systems since 2010 for various companies across the US. He currently lives in Austin, TX where he hides in the air conditioning for a large portion of the year.

Looking forward to another great meetup!

Please contact us if you would like to be a speaker for future events.

Location, food and drinks sponsored by Microsoft.