Inference review of vllm, sglang, gwen, kimi

There are loads of FE and GTM jobs out there from companies looking to get a foothold in the open source model business. Used to be servers in the golden age of the internet. Now it is GPU with models.

This session will cover a paper review of vllm, and sglang, Review the papers and replicate the benchmarks for a start.

LLMs are in the process of destroying the coordination type jobs. AKA the recent UBER layoffs.

The good part of LLMs is you can use them to your advantage. Find a repo, back out tickets and replicate a solution. Use the LLMs to figure out a learning path. They can ladder tickets from easy to hard.

Here is my first attempt:
vllm ticket review: https://github.com/vllm-project/vllm/issues/50128.
This ticket starts with comparing warm start vs. cold start. Someone found a path where there was a bug causing longer restarts because of tracing.
Speedgraph

https://github.com/dougc333/llm_benchmarking

There are 3 independent variables:

1. the client
2. the server, the ability for the server to process multiple requests from multiple clients dependent on #clients, #request/client, seq_len/client, and the num_cores and GPU. Prob too hard to sweep as a fn of different GPUs and num of cores but the vllm people say via twitter post somewhere most vllm configs are under configured for CPU.
3. the scheduler

The LLMs can devise a plan to measure all 3 and you can figure out what is real or not LLMs never say no so it may be necessary to run what the LLM says to get a sense of intuition. Errors are more valuable than the performance number. Focus on how the runtime works and the errors. Use the LLM to ladder github issues which are fixed by misconfiguration in vllm settings. Those you can use to help build experience quicker to support real customer tickets.