Ontology as a hyperparameter: Task-aware knowledge graph construction using LLMs

We invite you to the sixty seventh seminar of the monthly series of meetings conducted jointly by QFRG (Quantitative Finance Research Group) and DSLab (Data Science Lab) - University of Warsaw.

The meeting will be devoted to the topic: Ontology as a hyperparameter: Task-aware knowledge graph construction using LLMs, in which Jan Frąckowiak (FES UW) will discuss how knowledge graphs built from text can be used to create better predictive models. He will also show how, by using LLM-based agents, it is possible to automatically evolve the graph structure in response to prediction quality, paving the way for more flexible, task-oriented use of knowledge graphs in finance and other applications.

Presentation abstract:
The project investigates the construction of task-aware knowledge graphs (KGs) from text, treating graph structure as an optimization target rather than a fixed representation. Existing frameworks such as Neo4j-based pipelines and LLM-driven tools (e.g., LlamaIndex) focus on constructing large, general-purpose graphs without incorporating feedback from downstream tasks, despite evidence from evaluation frameworks (e.g., KGrEaT, GEval, KGBench) that KG utility is task-dependent. The research addresses this gap by proposing a feedback-driven approach in which ontology is treated as a tunable component influencing predictive performance. The study uses document-based datasets linking text to measurable outcomes, including financial news (FNSPID) paired with stock price movements. LLM agents are employed for ontology evolution and triple extraction, enabling scalable generation of multiple KG variants. Graph-derived features are then computed over temporal windows and used in predictive models, whose performance is fed back to the agents to refine ontology design, closing the loop between graph construction and downstream task. Preliminary results show that graph structure significantly impacts predictive performance, with adaptive ontology refinement leading to measurable improvements.