Dense Retrieval with Apache Solr Neural Search by Alessandro Benedetti

Alessandro Benedetti is the director of Sease and also an Apache Lucene/Solr PMC member and committer. He has contributed to the Neural Search in Apache Solr.

Neural Search is an industry derivation from the academic field of Neural information Retrieval. More and more frequently, we hear about how Artificial Intelligence (AI) permeates every aspect of our lives and this includes also software engineering and Information Retrieval.

In particular, the advent of Deep Learning introduced the use of deep neural networks to solve complex problems that could not be solved simply by an algorithm. Deep Learning can be used to produce a vector representation of both the query and the documents in a corpus of information. Search, in general, comprises of performing four primary steps:
- generate a representation of the query that describes the information need - generate a representation of the document that captures the information contained in it
- match the query and the document representations from the corpus of information
- assign a score to each matched document in order to establish a meaningful document ranking by relevance in the results

With the Neural Search module, Apache Solr is introducing support for neural network based techniques that can improve these four aspects of search.

This talk explores the first official contribution of Neural Search capabilities available from Apache Solr 9.0(may 2022):
Approximate K-Nearest Neighbor Vector Search for matching and ranking.

You will learn:
- how Approximate Nearest Neighbor (ANN) approaches work, with a focus on Hierarchical Navigable Small World Graph (HNSW)
- how the Apache Lucene implementation works
- how the Apache Solr implementation works, with the new field type and query parser introduced
- how to run KNN queries and how to use it to rerank a first stage pass Join us as we explore this new Apache Solr feature!