Kubernetes and nano-vllm Working Group
Details
With LLMs you can build a cluster in hours instead of weeks. The key is the verification loop where the LLM verifies the scaling up and scaling down behavior in a loop.
Pre LLM required weeks to get a config tested for production in small incremental steps. Beyond tedious.
PreLLM verification required hours or up to a complete day to get the config correct.
Current models do everything. Give it an instruction and it churns away to create a working demo.
Start with terraform, kubernetes, llama.cpp or nano-vllm. Use cpu instances or multiple L4 instances(T4 doesnt support bfloat16) if you can get them to build out a llm service to replace a company's frontier model token bill using Qwen.
If you are learning and this is new, start with a single hello world server in a pod and learn to autoscale up and down using kind, where all the pods are in a single process. Like hadoop with bigtop. Once scaling works in local and cloud then transition to Qwen in CPU. Dont bother with multi gpu setups. GPUs are impossible to reserve these days. Once CPU llama.cpp scaling is done then look to understand what kserve and llm.d do.
LLM Deployment and Kubernetes:
https://github.com/kserve/kserve
https://github.com/vllm-project/llm-compressor
https://github.com/llm-d/llm-d
https://github.com/GeeeekExplorer/nano-vllm
Since this is within the reach of LLMs, requiring where human employees are needed requires a rethink.