Open Data Platform and Data Observability by Acceldata

Hello everyone! Excited to invite you all to our 5th In-person meetup on July 4th at 11:00 AM, hosted by Acceldata in Bangalore. We’d love to see you there—come join us!
**********
📍 Venue:
Acceldata, Urban Vault, 1090/A, 18th Cross Rd, Sector 3, HSR Layout, Bengaluru- 560102
**********
***IMPORTANT***
Please ensure you obtain a “Visitor Badge” by presenting a government-issued ID (e.g., Aadhar Card, Driving License) at the Reception desk on the Ground Floor.
**********
Agenda:

• 11:00 - 11:10: Welcome
• 11:10 - 11:50: Vivek Pemawat, Principal Engineer, Acceldata
• 11:50 - 12:30: Snehasish Roy, Technical Lead, Phonepe
• 12:30 - 12:40: Break
• 12:40 - 13:20: Gowtham Sadasivam, Senior Staff Engineer, Acceldata
• 13:20 - 14:20: Lunch / Networking

****************
💡 Speaker: Vivek Pemawat, Principal Engineer, Acceldata
Talk: AI-Powered Automated Quality Engineering for Open Data Platform at Scale
Abstract:
The Open Data Platform is a complex ecosystem comprising more than 40 interconnected repositories built around core Hadoop technologies, including Apache Hadoop, Apache Hive, Apache HBase, Apache Ranger, and Apache Ambari. Maintaining quality, stability, and security across such a distributed platform presents significant engineering challenges, particularly when managing large-scale CVE remediation, dependency upgrades, and continuous feature delivery.

To address these challenges, we built an AI-powered, fully automated quality engineering framework that combines CI-based merge gating, nightly end-to-end validation, and intelligent test optimization. Every code change undergoes automated compilation, build verification, dependency checks, and quality validations before it can be merged, ensuring that unstable code never reaches shared integration branches. This significantly reduces build failures and improves development efficiency.
A dedicated nightly validation environment continuously deploys and tests the complete Hadoop ecosystem. The pipeline performs end-to-end functional validation, cluster health verification, upgrade testing, security validation, and cross-component integration checks across all platform services. This nightly quality gate provides continuous confidence that the platform remains stable despite ongoing development and security updates.
A key differentiator is the use of AI-driven test selection. By analyzing code changes and identifying impacted components, AI automatically selects the most relevant test suites instead of executing the entire regression pack. This reduces execution time, lowers infrastructure costs, accelerates feedback cycles, and enables faster release decisions while maintaining high quality standards.
The outcome is a scalable and intelligent quality engineering platform that automates security validation, accelerates CVE remediation, reduces manual effort, and improves release reliability across more than 40 repositories. By combining AI, automation, and continuous validation, the Open Data Platform has transformed quality assurance into a proactive, data-driven, and highly efficient engineering practice.
-------------------------------------------------
💡 Speaker: Snehasish Roy, Technical Lead, Phonepe
Talk: Clockwork: Building a Distributed Job Scheduler for 2 Billion Daily Callbacks
Abstract:
Have you ever had an alarm fail to wake you up, causing a ripple effect of chaos in your morning? At PhonePe, we understand the criticality of such 'alarms' in our digital ecosystem.
Take, for instance, our daily Merchant Settlements process. A merchant receives multiple transactions during the day. At the end of the day, we want to ensure the final amount gets credited to their account. A potential delay in this routine job being executed means a merchant not receiving their earnings on time, shaking their trust as a PhonePe Customer.

At PhonePe, we face the colossal task of managing over 2 billion daily callbacks. The ability to handle over 100,000 job schedules per second with single-digit millisecond latency is not just a goal; it's a necessity. At p99, our system ensures that there's no lag in job execution, which in the worst case is capped at 1 minute. Follow along to learn how we've made this possible.

In this talk, we will take a look at the internals of Clockwork – the system that powers job scheduling across various teams at PhonePe and enables clients to easily onboard and schedule future jobs without the need for heavy lifting on their own.
---------------------------------------------
💡 Speaker: Gowtham Sadasivam, Senior Staff Engineer, Acceldata
Talk:
TBC
Abstract:
TBC