The Israeli Natural Language Processing Meetup (NLP IL)

For the next meeting of the Vision-Language Club we are very excited to host Idan Schwartz for a talk on Discriminative Models Can Make Generative Models Better.

We will meet on the 5th of May at 20:00 online. The talk will also be recorded and uploaded to NLP-IL's YouTube channel. The talk will be conducted in Hebrew.

In recent years, the exponential growth in data and model sizes has led to significant advancements in generative language and image models. Despite these improvements, generative models often require fine-tuning to adapt to specific domains or tasks. This fine-tuning typically involves a curated dataset and an additional, computationally intensive learning phase.
In this talk, I will propose an alternative approach that guides the generation process during inference by employing an external discriminative model. The method iteratively generates outputs and uses the external model to evaluate and optimize the results. This process repeats until a satisfactory output is achieved.

I will demonstrate the effectiveness of this method for both language models and diffusion-based text-to-image models. Specifically, I will show how language models can support multimodal input through the use of CLIP, and for image generation, how to adapt to fine-grained domains (e.g., rare animal species), improve object-count accuracy in generated images with a counting-model approximation, and support subject-driven generation.

Idan Schwartz is an Assistant Professor at Bar-Ilan University. His academic work focuses on multimodal learning. He holds a Ph.D. in Computer Science from the Technion, where he collaborated with Prof. Tamir Hazan and Prof. Alexander G. Schwing (UIUC), and previously served as a postdoctoral researcher in Prof. Lior Wolf’s Deep Learning lab.In industry, Idan held research roles at Microsoft and eBay. He later served as Head of Research at Spot by NetApp and is currently working on a new project.