Houston Machine Learning

Text-to-SQL for Enterprise Data Analytics

The introduction of large language models has brought rapid progress on Text-to-SQL benchmarks, but it is not yet easy to build a working enterprise solution. In this paper, we present insights from building an internal chatbot that enables LinkedIn’s product managers, engineers, and operations teams to self-serve data insights from a large, dynamic data lake. Our approach features three components.

First, we construct a knowledge graph that captures up-to-date semantics by indexing database metadata, historical query logs, wikis, and code. We apply clustering to identify relevant tables for each team or product area.

Second, we build a Text-to-SQL agent that retrieves and ranks context from the knowledge graph, writes a query, and automatically corrects hallucinations and syntax errors.

Third, we build an interactive chatbot that supports various user intents, from data discovery to query writing to debugging, and displays responses in rich UI elements to encourage follow-up chats. Our chatbot has over 300 weekly users. Expert review shows that 53% of its responses are correct or close to correct on an internal benchmark set. Through ablation studies, we identify the most important knowledge graph and modeling components, offering a practical path for developing enterprise Text-to-SQL solutions.

Slides for past meetups posted: Github
Recordings have been posted at: YanAITalk

Feel free to reach out if you want to present a paper or a use case at upcoming meetups!

Note: You must have a Zoom account to login (free account is sufficient). Link and password will be shared three days before the meeting.

We are going to review the different views of Small language models (SLMs) in the world of Agentic AI:
Small Language Models are the Future of Agentic AI
Large language models (LLMs) are often praised for exhibiting near-human performance on a wide range of tasks and valued for their ability to hold a general conversation. The rise of agentic AI systems is, however, ushering in a mass of applications in which language models perform a small number of specialized tasks repetitively and with little variation. Here we lay out the position that small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI. Our argumentation is grounded in the current level of capabilities exhibited by SLMs, the common architectures of agentic systems, and the economy of LM deployment. We further argue that in situations where general-purpose conversational abilities are essential, heterogeneous agentic systems (i.e., agents invoking multiple different models) are the natural choice. We discuss the potential barriers for the adoption of SLMs in agentic systems and outline a general LLM-to-SLM agent conversion algorithm.

Slides for past meetups posted: Github
Recordings have been posted at: YanAITalk

Feel free to reach out if you want to present a paper or a use case at upcoming meetups!

Note: You must have a Zoom account to login (free account is sufficient). Link and password will be shared three days before the meeting.

A Multi-Agent Collaboration Framework for Complex IT Query Support

This paper presents a multi-agent collaboration framework (MACF) powered by large language models (LLMs) for handling complex information technology (IT) support and technical queries. Our system implements a hierarchical workflow that decomposes user queries into manageable sub-tasks, orchestrates multiple specialized agents for parallel execution, and synthesizes their outputs into concise and clear responses. The framework features four key components: a planner node for query decomposition and agent selection, an execution node managing parallel sub-agent operations, a summarization node for result consolidation, and an output node for response generation. We incorporate human-in-the-loop feedback mechanisms and support interactive follow-up conversations to ensure accuracy and user satisfaction. To evaluate the planner’s accuracy and effectiveness of the workflow, we build an expert grounded complex IT Q&A dataset that includes 100 question and answer pairs. Four metrics were evaluated in the experiment, planner accuracy evaluated by human expert, helpfulness, clarity and factual accuracy evaluated by LLM respectively. Experimental results demonstrate that the framework effectively handles a wide range of technical support scenarios with fast and efficient execution.

Slides for past meetups posted: Github
Recordings have been posted at: YanAITalk

Feel free to reach out if you want to present a paper or a use case at upcoming meetups!

Note: You must have a Zoom account to login (free account is sufficient). Link and password will be shared three days before the meeting.