Agentic AI Use Case: A Multi-Agent Collaboration Framework for Complex IT Query

A Multi-Agent Collaboration Framework for Complex IT Query Support

This paper presents a multi-agent collaboration framework (MACF) powered by large language models (LLMs) for handling complex information technology (IT) support and technical queries. Our system implements a hierarchical workflow that decomposes user queries into manageable sub-tasks, orchestrates multiple specialized agents for parallel execution, and synthesizes their outputs into concise and clear responses. The framework features four key components: a planner node for query decomposition and agent selection, an execution node managing parallel sub-agent operations, a summarization node for result consolidation, and an output node for response generation. We incorporate human-in-the-loop feedback mechanisms and support interactive follow-up conversations to ensure accuracy and user satisfaction. To evaluate the planner’s accuracy and effectiveness of the workflow, we build an expert grounded complex IT Q&A dataset that includes 100 question and answer pairs. Four metrics were evaluated in the experiment, planner accuracy evaluated by human expert, helpfulness, clarity and factual accuracy evaluated by LLM respectively. Experimental results demonstrate that the framework effectively handles a wide range of technical support scenarios with fast and efficient execution.

Slides for past meetups posted: Github
Recordings have been posted at: YanAITalk

Feel free to reach out if you want to present a paper or a use case at upcoming meetups!

Note: You must have a Zoom account to login (free account is sufficient). Link and password will be shared three days before the meeting.