[UCL WI Talks]: Evaluating Automated Citation Screening in Systematic Reviews

Evaluating Automated Citation Screening in Systematic Reviews: Metrics, Review Outcomes, and Datasets

Abstract:
This talk will discuss the evaluation of automated citation screening for systematic literature reviews. As systematic reviews require detailed examination of vast amounts of literature, many researchers are turning to automated tools to help with the screening process. But how effective are these tools?
First, we will examine the Work Saved over Sampling (WSS) metric, commonly used to judge how well these automated tools work. We will highlight some issues with WSS and propose alternative metrics. Next, we’ll explore how focusing on the actual outcomes of systematic reviews can provide a more accurate assessment of automation’s effectiveness. We will present findings from an outcome-based evaluation framework, emphasising a shift from aiming for 100% Recall to aligning with the quality of review outcomes.
Lastly, we’ll tackle the topic of datasets used for screening. The choice of the dataset can significantly influence how effective the automated screening model is. We will offer some insights on how to make better dataset choices and introduce CSMeD, a meta-dataset consolidating over 300 systematic reviews. We will conclude with perspectives on the future trajectory of screening evaluation, especially concerning large language models.

Bio:
Wojciech is a Research Assistant and a final-year PhD student at TU Wien, focusing on automating the Systematic Literature Reviews process. He is a member of the EU Horizon 2020 Project DoSSIER, specialising in biomedical natural language processing. His research interests include information extraction, evaluation, and language modelling. Wojciech holds degrees in Computer Science from AGH University and Cognitive Science from Jagiellonian University in Cracow, Poland. He has previously worked as an NLP Research Engineer at Samsung and interned at Sony, UNINOVA R&D, and the Polish Academy of Sciences.