38th Meet up: Using On-Premise LLMs to Control Robots

Despite rapid advancements in robotics and AI, most robots today are not optimized for local, natural language interaction. Their reliance on cloud-based AI services introduces critical limitations, ranging from privacy and security concerns to poor reliability in remote environments and high latency during deployment. With the rise of compact, high-performance edge devices and increasingly lightweight language models, we are at a turning point in robotics.

In this talk, I’ll share how we built an offline, edge-based system that allows robots, from drones to a quadruped robot dog, to understand and execute natural language instructions using local LLMs running on Jetson hardware. I’ll walk through how we fit an LLM into a constrained architecture, built autonomy without the internet access, and then used that LLM to actively control a robot in real time.

We’ll close by exploring the road ahead: a future where robots are fully agentized, responsive to beginners using everyday language, and no longer dependent on the cloud. Imagine giving commands to every robot in your life, with words alone.