Probing Ranking LLMs: Mechanistic Interpretability in Information Retrieval

SEA XL: James Allan will join us on location to present his views on Probing Ranking LLMs: Mechanistic Interpretability in Information Retrieval.

This is a hybrid event. The speaker and on-site audience can be found at Lab42, Room L3.33. The Zoom link is visible once you "attend" the meetup on this page.

Speaker: James Allan, Center for Intelligent Information Retrieval, University of Massachusetts Amherst
Title: Probing Ranking LLMs: Mechanistic Interpretability in Information Retrieval
Abstract: Transformer networks, especially those with performance on par with GPT models, are renowned for their powerful feature extraction capabilities. However, the nature and correlation of these features with human-engineered ones remain unclear. In this study, we delve into the mechanistic workings of state-of-the-art, fine-tuning-based passage-reranking transformer networks.
Our approach involves a probing-based, layer-by-layer analysis of neurons within ranking LLMs to identify individual or groups of known human-engineered and semantic features within the network's activations. We explore a wide range of features, including lexical, document structure, query-document interaction, advanced semantic, interaction-based, and LLM-specific features, to gain a deeper understanding of the underlying mechanisms that drive ranking decisions in LLMs.
Our results reveal a set of features that are prominently represented in LLM activations, as well as others that are notably absent. Additionally, we observe distinct behaviors of LLMs when processing low versus high relevance queries and when encountering out-of-distribution query and document sets. By examining these features within activations, we aim to enhance the interpretability and performance of LLMs in ranking tasks. Our findings provide valuable insights for the development of more effective and transparent ranking models, with significant implications for the broader information retrieval community. All scripts and code necessary to replicate our findings are made available.
Bio: James Allan is a Professor in the Manning College of Information and Computer Sciences at the University of Massachusetts Amherst where he also serves as Associate Dean of Research. He directs the Center for Intelligent Information Retrieval (CIIR), a very well-known academic research lab in Information Retrieval (IR). His research focuses on ways to process, organize, retrieve, and browse unstructured text information, and he is particularly interested in using language tools to support critical thinking by readers. His other major research activities look at ways to recognize contention and misinformation in text, to support effective interactive search techniques, and to build and evaluate more effective search models on large collections of within- and cross-language text.

Counter: Just keep going! This is SEA Talk #273.