Building Scalable Data Pipelines with Kafka and Apache Spark

Hello,

Let's meet and discuss building data pipelines using Kafka and Spark.
This talk discusses the process of building data pipelines, e.g. extraction, cleaning, integration, pre-processing of data, in general all the steps that are necessary to prepare your data for your data-driven product. In particular, the focus is on data plumbing and on the practice of going from prototype to production.

Some of the topics we will cover include:

1. What are Data Pipelines
2. Use cases for Data Pipelines
3. What does it mean for data pipelines to be scalable?
4. A generic data pipeline architecture
5. What is Kafka and where it stands in the data pipeline architecture
6. Kafka architecture and components
7. Example of Kafka client API usage (Producer, Consumer and Connect API)
8. Walkthrough of codebase and Kafka API
9. What Spark is and where it stands in the architecture
10. Spark architecture and components
11. An example of a Spark job (reading from Kafka, perform some use case driven computation, write to a data store like Redis)
12. Walkthrough of codebase and Spark API
13. QA

Speaker:
Babatunde Ekemode - Software Engineer

Is a Software Engineer at Africastalking Ltd. He is passionate about distributed systems - with a primary focus on big data analytics tools and microservices architecture and DevOps. Over a period of 8 years, Tunde has built his career and experience building software product in e-commerce, chips technology, travel tech space. He started out with Java, then stumbled upon Scala and he has stayed glued ever since. Tunde enjoys collaborating with a team of focused and enthusiastic professionals on software products design, implementation and maintenance. Unproductive meetings give him the blues. Tunde is Football fanatic and Ole is the new Mr Fantastic.