Leveraging Microservices & Kafka to Scale Developer Productivity

Join us for an Apache Kafka meetup on February 12th from 6:00pm, hosted by Yugabyte. Address, agenda and speaker information can be found below. See you there!

---

Agenda:
6:00pm - 6:30pm: Networking, Pizza and Drinks
6:30pm - 7:15pm: Mohinish Shaikh, Confluent
7:15pm - 8:00pm: Bogdan Matican and Bharat Baddepudi, Yugabyte
8:00pm - 8:30pm: Additional Q&A and Networking

---

Talk 1:

Speaker:
Mohinish Shaikh, Engineer at Confluent

Title:
Leveraging Microservices & Kafka to Scale Developer Productivity

Abstract:
Developer productivity is a discipline that has increasingly become critical for companies and organizations to deliver software reliably, on time and with reasonable cost.

Microservices is an approach to application development in which a large application is built as a collection of modular, isolated components or services. Organizations are increasingly adopting microservices architecture to shorten time to market of a software product as it helps optimizing teams to be more productive by implementing automation throughout the life-cycle of a product.

Leveraging microservices architecture for the management and automation of developer productivity pipelines helps enhance organization's productivity at scale. In this talk, I will share my experiences leveraging microservices and kafka to build automated end-to-end self serviceable productivity pipelines that enable build, test, deployment, release, packaging, orchestration, provisioning, observability and chaos engineering systems.

In this context, Kafka helps with aggregation of valuable insights on these independent services as they collaborate with each other in real-time, which is then leveraged to learn and enhance the infrastructure platform.

Developer productivity systems uses Kafka as a enterprise message bus to pass context, data and configuration to trigger complex interplay among various tools, services, infrastructure and environments as code propagates from a developer’s machine to production. In addition, data is harvested from monitoring, tracing, observability frameworks to be routed through Kafka to generate real time state of the propagation using streaming analytics to paint a correlation between various phases of software lifecycle. This real time data is also fed into learning systems using AI/ ML to learn about code propagation lifecycle, failures across services and resource utilization of the clusters across various environments.

Talk 2:

Speakers: Bogdan Matican and Bharat Baddepudi, Founding engineers at Yugabyte

Title: Best Practices for Running Streaming Apps on Kubernetes

Abstract: Kubernetes has become the de-facto standard for orchestrating stateless microservices on containerized infrastructure. However, the same level of maturity is missing when it comes to Kubernetes-driven orchestration of business-critical stateful data services. Streaming apps and data pipelines powered by Kafka and KSQL are perfect examples of such services. In this talk, we will highlight a few best practices for operating streaming apps at scale in Kubernetes environments. Specifically, we will discuss how to map the various components of streaming apps to Kubernetes controllers (including StatefulSets), StatefulSet configuration for high performance and data resilience with local persistent volumes and pod anti-affinity as well as multi-zone/multi-region deployments for tolerating cloud provider failures. We will demonstrate many of these aspects using a sample IoT app, built on the CKY (Confluent Kafka + KSQL + YugaByte DB) stack, deployed on Kubernetes.

---

Don't forget to join our Community Slack Team! https://launchpass.com/confluentcommunity