Conformal Prediction under Ambiguous Ground Truth

Uncertainty estimation is crucial in many areas, from medical applications to self-driving cars and weather forecasting, to allow the widespread use of machine learning models.

We are excited to have David Stutz, a research scientist at Google DeepMind, in our joint Heidelberg.ai / NCT Data Science Seminar series.
In this online seminar, David Stutz will talk about conformal prediction, a method to give rigorous uncertainties to machine learning models, and how to extend it to cases where even the ground truth data is uncertain.

We look forward to your participation, as this seminar will equip us with the knowledge to enhance the safety of our machine-learning models, making them even more reliable.

Thursday, Mai 23rd, 4 pm
Zoom
Event details:
Heidelberg.ai

Abstract
In safety-critical classification tasks, conformal prediction allows to perform rigorous uncertainty quantification by providing confidence sets including the true class with a user-specified probability. This generally assumes the availability of a held-out calibration set with access to ground truth labels. Unfortunately, in many domains, such labels are difficult to obtain and usually approximated by aggregating expert opinions. In fact, this holds true for almost all datasets, including well-known ones such as CIFAR and ImageNet. Applying conformal prediction using such labels underestimates uncertainty. Indeed, when expert opinions are not resolvable, there is inherent ambiguity present in the labels. That is, we do not have ``crisp'', definitive ground truth labels and this uncertainty should be taken into account during calibration. In this paper, we develop a conformal prediction framework for such ambiguous ground truth settings which relies on an approximation of the underlying posterior distribution of labels given inputs. We demonstrate our methodology on synthetic and real datasets, including a case study of skin condition classification in dermatology.