Everything on Hard Mode: GPUs on Kubernetes

Speaker
Clayton Yochum

Description
Adding a GPU resource request is the easy part. Getting a production inference service to start quickly, scale usefully, survive hardware faults, and shut down safely is...less easy. It's still Kubernetes, but on hard mode.

Drawing on experience building and operating large-scale ML systems at Abridge, Clayton will follow a GPU workload through hardware selection, model loading, traffic management, and failure recovery. He'll tell you how he screwed up at nearly every step, so maybe you don't have to.

Familiarity with Kubernetes is helpful; an ML background is not required.

About the Speaker
Clayton is a machine learning engineer focused on production ML systems. As an early ML Engineer at Abridge, he built and operated ML inference infrastructure on Kubernetes, working across across backend, ML, and platform teams. He likes spending someone else's money on GPUs and making them go brrrr.

If you would like to present at a future meetup, fill out this form or email utah.k8s@gmail.com.