Best Practices and Lessons Learned from Enterprise Level MSK (Kafka) deployment

Talk description:

Transaction Banking from Goldman Sachs is a high volume, latency sensitive digital banking platform offering. We have chosen an event driven architecture to build highly decoupled and independent microservices in a cloud native manner and are designed to meet the objectives of Security, Availability Latency and Scalability. AWS MSK (Managed Kafka) was a natural choice – to decouple producers and consumers and to scale easily for high volume processing. However, there are certain aspects that require careful consideration – handling errors and partial failures, managing downtime of consumers, secure communication between brokers and producers / consumers. In this session, we will present the patterns and best practices that helped us build robust event driven applications. We will also present our solution approach that has been reused across multiple application domains. We hope that by sharing our experience, we can establish a reference implementation that application developers can benefit from.

About the Speakers:

Aruna Kalagnanam is a Tech Fellow in the Transaction Banking Group in Goldman Sachs developing security solutions in the Platform Engineering Team. Aruna enjoys working on distributed systems for the cloud and her past experience has been primarily in developing application frameworks, API management platforms and solutions for Cloud platforms.

Gayatri is a Vice president in Transaction Banking at Goldman Sachs where she heads Business Activity Monitoring globally. She has 18+ years of experience in software development, performance engineering, testing and production management in the finance and web domains. Gayatri is a AWS Community Builder in the Management, Governance and Migration track