Customizing and Evaluating LLMs, an Ops Perspective

if your product or domain isn't fully generic, you'll probably hit a lot of challenges that make deploying your LLM application a meaningful risk.

In this talk, I'll discuss the main challenges in customizing and evaluating LLMs for specific domains and applications, and suggest a few workflows and tools to help solve for those challenges.

With LLM hype growing ever greater, almost every company is racing to create their LLM application, whether it's an internal tool to boost productivity, or a chat interface for their product.