SEA April: Uncertainty Estimation for IR and NLP (hybrid)

***
IMPORTANT: You will be able to view the Zoom link once you 'attend' the meetup on this page. The physical event will be at Science Park in Amsterdam, room SP C0.110.
***

After the first hybrid edition of SEA in March, we continue to organize SEA in a hybrid fashion in April. This edition is all about uncertainty estimation for IR and NLP.

17.00: Carsten Eickhoff (Brown University).

Title: Uncertainty and Calibration in Search

Abstract: Recent work in neural information retrieval models have achieved impressive performance on a variety of retrieval tasks whether the models are based on pre-trained Transformer architectures or learned from scratch. These state-of-the-art models treat their estimates of a document's relevance as a deterministic score. While effective, such a deterministic perspective obfuscates a large amount of critical information that a user could use to determine whether their query is effective or when they have gone so far down a ranked list that the model is no longer sure of its scores. In this talk I will present an efficient uncertainty modeling scheme for neural retrieval models and demonstrate its merit in biomedical query performance prediction.

Bio: Carsten is an Assistant Professor of Computer Science at Brown University. His lab focuses on improving the effectiveness and transparency of natural language processing and information retrieval techniques and assessing their translational merit in the biomedical domain. Prior to joining Brown, he graduated from The University of Edinburgh and TU Delft, and was a postdoctoral fellow at ETH Zurich and Harvard University. Carsten has authored more than 100 articles in computer science conferences (e.g., SIGIR, ACL, EMNLP, NAACL, WWW, KDD, WSDM, CIKM) and clinical journals (e.g., Nature Digital Medicine, The Lancet - Respiratory Medicine, Radiology, European Heart Journal). His research has been supported by the NSF, NIH, DARPA, IARPA, Google, Amazon, Microsoft and others. Aside from his academic endeavors, he is a founder and board member of several deep technology startups in the health sector that strive to translate technological innovation to improved safety and quality of life for patients.

17.30: Jiahuan Pei (Amazon).

Title: Transformer Uncertainty Estimation with Hierarchical Stochastic Attention

Abstract: Transformers are state-of-the-art in a wide range of NLP tasks and have also been applied to many real-world products. Understanding the reliability and certainty of transformer model predictions is crucial for building trustable machine learning applications, e.g., medical diagnosis. Although many recent transformer extensions have been proposed, the study of the uncertainty estimation of transformer models is under-explored. In this talk, I will introduce the stochastic transformers, which have been published at AAAI 2022 conference. We enable transformers with the capability of uncertainty estimation and, meanwhile, retain the original predictive performance. This is achieved by learning a hierarchical stochastic self-attention that attends to values and a set of learnable centroids, respectively. Then new attention heads are formed with a mixture of sampled centroids using the Gumbel-Softmax trick. We theoretically show that the self-attention approximation by sampling from a Gumbel distribution is upper bounded. We empirically evaluate our model on two text classification tasks with both in-domain and out-of-domain datasets.

Lots of numbers still left: Talks #203 and #204!