Borealis AI Lecture Series - Prof. Leonid Sigal - CV in the Age of LLMs

This edition of our Lecture Series is intended for university faculty and graduate students in machine learning across computer science, ECE, statistics, mathematics, as well as industry practitioners in researcher, engineering etc. in the GTA.

Talk: Computer Vision in the Age of Large Language Models (LLMs)

Abstract: The use of large language models (LLMs) in computer vision has grown rapidly over the past few years, leading to a significant shift in the field. In this talk, Prof. Sigal will discuss recent research conducted by Prof. Sigal's team at UBC that explores this evolving landscape. Prof. Sigal's focus will be on three areas of research. Firstly, Prof. Sigal will delve into Prof. Sigal's team's recent work (ICLR'23) on building foundational image representation models by combining two successful strategies of masking (e.g., BERT) and sequential token prediction (e.g., GPT). Their findings suggest that this combination results in a better, more efficient, and transferable pre-training strategy.
Secondly, Prof. Sigal will discuss a series of papers that focus on text-to-image generative models. They introduce a novel autoregressive diffusion-based framework with a visual memory module that implicitly captures the actor and background for multi-frame story visualization (CVPR'23). This design maintains consistency and resolves references in longer story text. Prof. Sigal will also briefly touch upon their recent (unpublished) work on controllability of such models through textual prompt inversion (accepted to CVPR'24) and example-based personalization (in submission). Thirdly, Prof. Sigal will discuss their work on biases in such models and their research on bias quantification in TTI models (also in submission). If time permits, Prof. Sigal will briefly describe their most recent work on continual object detection. They introduce a memory-based detection transformer architecture to adapt a pre-trained DETR-style detector to new tasks while preserving knowledge from previous tasks. Their proposed model outperforms state-of-the-art by significant margins, with a novel localized query function for efficient information retrieval from memory units, aiming to minimize forgetting.

Agenda:
*All times EST
12:00 pm - 12:05 pm - Arrival
12:05 pm - 12:10 pm - Introduction of Speaker by Borealis AI's Director of Research Partnerships
12:10 pm - 12:50 pm - Lecture by Prof. Leonid Sigal
12:50 pm - 1:00 pm - Audience Q&A