Productionalizing ML at Scale with Ray Serve

Welcome to our Ray meetup, where we focus on Ray and Ray’s native libraries for scaling machine learning workloads. We will discuss Ray Serve, an ML framework-agnostic, production-ready, operational, and scalable model serving library for this meetup.

The talks will cover three functional areas of model serving:

• An overview of Ray Serve features and functionality and roadmap
• On building multi-model inference graphs with Ray Serve and scaling with Ray
• Operationalizing Ray Serve

To tie all these aspects of Ray Serve together, we will show a demo.

Agenda:

• 6:00 PM Welcome remarks, Announcements & Agenda by Jules Damji, Anyscale
• 6:05 PM "Ray Serve: Overview and roadmap," Edward Oakes, Simon Mo Anyscale
• 6:15 PM Q & A
• 6:20 PM "Developing and deploying scalable multi-model inference graphs," Jiao Dong, Anyscale
• 7:00 PM Q & A
• 7:10 PM "Operationalizing Ray Serve," Shreyas Krishnaswamy, Anyscale
• 7:40 PM Q & A
• 7:45 PM Demo Q & A

Join us if you are interested in serving and operationalizing ML models at scale using Ray Serve!

Talk 1: Ray Serve: Overview and future roadmap

In this introductory session, we’ll discuss the motivation behind Ray Serve, who’s using Ray Serve and why, recent features and updates, and look to the future feature roadmap as we approach Ray 2.0.

Bio: Edward Oakes is a software engineer and project lead on the Ray Serve team. He works across the stack at Anyscale, from Ray core to Ray Serve to the Anyscale platform.

Talk 2: Developing and deploying scalable multi-model inference graphs

In this talk, we aim to show how to leverage the programmable and general-purpose distributed computing ability of Ray to facilitate the authoring, orchestrating, scaling, and deployment of complex serving graphs as a DAG under one set of APIs, like a microservice, so a user can program multiple models dynamically on your laptop as if you’re writing a local python script, deploy to production at scale, and upgrade individually.

Bio: Jiao Dong is a software engineer focusing on Ray Serve and Ray infrastructure at Anyscale.

Talk 3: Operationalizing Ray Serve

In this session, we will introduce you to a new declarative REST API for Ray Serve, which allows you to configure and update your Ray Serve applications without modifying application files. You can incorporate this API into your existing CI/CD process to manage applications on Ray Serve as part of your MLOps lifecycle.

Bio: Shreyas Krishnaswamy is a software engineer focusing on Ray Serve and Ray infrastructure at Anyscale.