How useful are results from simulated test collections? by David Hawking

Prof. David Hawking, ANU and The University of Melbourne, reporting
joint work with Bodo Billerbeck, Nick Craswell, and Paul Thomas, all from Microsoft.

SPEAKER
Prof. David Hawking (ANU & The University of Melbourne)

ABSTRACT
Cloud companies which host document data on behalf of
clients wish to provide effective and responsive indexing and
retrieval services for their clients, but staff from the cloud company
are usually prevented from looking at client data, queries, and click
logs. Microsoft examples include Exchange 365 and the Azure Cloud.
The open source SynthaCorpus system provides techniques for
statistically emulating a corpus and a query log, as well as for
generating corpus-appropriate known-item test sets.

But how well do results from a simulated collection predict the real
performance for the client? We put SynthaCorpus methods to the test
using three open source search systems (Indri, Terrier and ATIRE, each
using a different ranking model) and four TREC datasets (AP, FR,
Patents, and WT10g). We also investigate the trade-off between
privacy and predictive accuracy by including two forms of emulation by
encryption. Finally, to investigate the influence of "noise" we report
predictions for the /bin/cp emulation method.

My opinion is that the findings show that there are simulation methods
which are capable of making predictions accurate enough for practical
use while adequately preserving privacy. However, in reality, that
decision would be in the hands of clients. We found that there is
indeed a trade-off between privacy and prediction accuracy and that
there are interesting interactions between retrieval system and
dataset.

BIO
In the 1990s, while at ANU, David Hawking was a co-ordinator of
the TREC Very Large Collection and Web Tracks, and co-creator of a number of well-known IR test collections. He was an IR researcher at
CSIRO for ten years and his research was spun off into the Funnelback
enterprise and web search company, where he worked as Chief Scientist.
In 2013 he joined Microsoft, working on the Bing search engine until
his retirement in 2018. Since retiring he has retained honorary
positions at ANU and The University of Melbourne. Recreational travel
permitting, he continues to work on the (to him) fascinating problem
of test collection simulation.

NOTE: Food and drinks will not be provided for this event.