Kubernetes and nano-vllm Working Group
Details
With LLMs you can build a cluster in hours instead of weeks.
This was not the case a year ago. We are seeing increasing layoffs as a result of the zero shot capabilities of LLMs. https://www.youtube.com/watch?v=tt_SloZ18Mk
There have been SWE layoffs particularly focused towards API programmers since all their work can be done with a LLM. The API surfaces are turned into verifable RL environments and they can train a LLM to do the work faster than a human. The opposite side is there has also been a rise in demand for an AIInfra engineers to serve the models and queries. Here is an example: https://jobs.ashbyhq.com/pika/35eb68a1-c26b-4221-80e6-51e21ec8edd9. There are too many of these jobs to count.
Everybody wants the sweet spot of 5y but with the quality of LLMs it is possible to accelerate that with personal demos you can use to convice a hiring manager you are ready for live traffic.
This is a working group. There is no formal lecture or teaching format. Do your own demos like it is work and build a set of demos.
LLM Deployment and Kubernetes:
https://github.com/kserve/kserve
https://github.com/vllm-project/llm-compressor
https://github.com/llm-d/llm-d
https://github.com/GeeeekExplorer/nano-vllm
https://github.com/dougc333/terraform
Intro Video with K8 autoscaling and a macbook serving llama.cpp python coding question queries. Since GPUs are largely sold out across all clouds we can substitute with CPU instances.
https://youtu.be/uo6Ckpr-iuQ
Kube Pipeline Example on Azure Devops. https://www.youtube.com/watch?v=Alm0SXbwris
Group Exercise: The Joy of React and CSS from Josh. https://www.joyofreact.com/. We need 6-7 people to come up with a multimodal benchmark and dataset we can use to fine tune a multimodal LLM.