AI-Driven Autoscaling: Adding Pods Before the Spike, Not After

If you run services on Kubernetes, you've met the Horizontal Pod Autoscaler and its one big idea: watch CPU, add pods when it climbs, drop them when it falls. That sounds fine until you're watching latency spike while CPU sits at a comfortable 25 percent. CPU is rarely the thing that's hurting you. On a JVM service the real pressure is usually heap, GC pauses, a saturated thread pool, or a pod that came up 20 seconds ago and still can't serve a request because it's warming up. HPA sees none of that, so it reacts late, overshoots, and then churns pods up and down while your p95 suffers and your bill doesn't budge.

This talk is what came out of trying to do better: replacing threshold rules with an autoscaler that learns from real traffic history instead of guessing from one metric. I'll cover why reactive scaling fights you, what it means to treat scaling as a prediction problem (add capacity at 8:55 for the 9 AM Monday rush, not at 9:05 once the queue is already backed up), and the results we saw: latency down roughly 23 percent, about a fifth off the cost, and oscillations cut from fifteen an hour to three. I'll also be straight about the painful parts. A learned scaler needs weeks of traffic before it's any good, it will confidently make the wrong call the first time it meets a Black Friday, and you want firm guardrails plus a boring fallback to plain HPA for when it misbehaves.

Audience will leave knowing when this is worth the effort, when it really isn't, and a few ideas for making even a stock autoscaler smarter about the signals it watches.

Speaker
Shalini Sudarsan is a DevOps Engineering Leader at KinderCare Learning Companies, USA. designing reliable, secure, and cost-optimized data and AI platforms. A Forbes Technology Council Member, Fellow of IETE and Women in Engineering (WIE) Oregon section, she drives enterprise AI adoption with a governed operating model that speeds time-to-market while lowering risk and spend. Shalini’s expertise spans BI strategy, data platform architecture, MLOps, observability, and value realization. She is known for translating complex engineering into measurable business outcomes. Shalini brings deep technical rigor and business expertise in the areas of DevOps and Reliability Engineering.A committed advocate for advancing technology, Shalini regularly presents at international conferences and contributes to IEEE and ACM as a technical reviewer.