Multimodal Search and Feature Exploration

This is a past event

41 people went


This Friday, we'll have two talks followed by drinks. The industry talk will be given by Gerbert Kaandorp from ( Roeland Ordelman from University of Twente will give the academic talk. The titles and abstracts will be announced later.

Please not that this meetup will be held in Universiteitsbiblioth­eek.


16:00 - 16:30 Roeland Ordelman

16:30 - 17:00 Gerbert Kaandorp

17:00 - 18:00 Drinks & Snacks

Details of the talks:


Roeland Ordelman--Multimodal Search in Video Hyperlinking (Video-to-Video Search)

Abstract: Nowadays constantly growing amount of multimedia content is produced by professionals and non-professionals on various types of equipment. This data is further shared or stored in multimedia repositories. One of the research challenges in this context is to develop algorithms that would allow users to explore this multimedia without prior knowledge of the collection, and to learn how to create personalised narratives through a network of linked video segments for different users. We call the process of creating this linked network between videos - video hyperlinking (VH). In order to evaluate the progress of algorithm development, we set up and run VH evaluations at the MediaEval [masked]) and TRECVid (2015-present)Benchmark evaluation workshops. The video hyperlinking task asks participants to return relevant video segments of arbitrary length (link targets) given query video segments that we call ‘anchors’. The video hyperlinking (VH) task can be seen as a video-to-video search task: given a video fragment, VH systems provide relevant other video fragments that are topically related, while not being (near) duplicates.

In this talk I highlight the details of the Hyperlinking task framework in 2016, and focus on different approaches that were implemented by task participants. I will outline the successful methods, and discuss questions raised by evaluation of the results.


Gerbert Kaandorp--Creating Meaningful Labeled Data by Interactive Feature Exploration

Abstract: In many applications today we are confronted with large amounts of unlabeled, or poorly labeled data. Our extensive product databases are no exception. In this talk we present our current work on the design of an interactive method for the creation of new and meaningful labels.
Our method relies on the interaction between AI algorithms like clustering and neural network feature generators on the one hand, and human verification of intermediate results on the other.We define a work cycle in which poorly labeled data is processed by neural networks in order to produce image embeddings in the form of feature vector representations, followed by clustering and dimension reduction of the feature data.
Both clustering and dimension reduction are a form of discovery, where clustering finds new candidate categories and dimension reductions allows us to focus on the relevant parts of the feature data. In particular, reducing the dimensions of the feature data withing each candidate category allows us to sort products according to novel criteria. The amount of data under consideration also makes dimension reduction a necessary condition if we want to boost search performance.
We discuss our methods for inspecting the resulting clusters and their influence on the quality of search results. The interactive aspect of this approach lies in the human verification of the new discoveries and, where meaningful, the application of new labels to the product data.
Ultimately, the cycle restarts when we use new labels in the next round of neural network training. This new generation of models can provide is with yet more meaningful image embeddings of our products. In turn, this allows a next phase of discovery by clustering, etc.
By iterating this process we aim to create new and meaningful categories in our product databases and improve the classification accuracies of our neural network models. This means we can provide our customers with a powerful tool to explore their product data and obtain more informative search results.

Gerbert Kaandorp is a data scientist in He studied Artificial Intelligence at the University of Amsterdam way back in the nineties. He founded several companies including Backbase.