Name: Knowledge Distillation, Model Ensemble and Its Application on Visual Recognition
Start: 2022-01-24T19:00:00+02:00
End: 2022-01-24T20:30:00+02:00

This talk will cover the principles and mechanisms of knowledge distillation (KD), as well as several KD framework designs for practical usage, including (1) How KD works and its relationship to Label Smoothing technique; (2) Model ensemble using knowledge distillation for improving the performance; (3) An application of knowledge distillation for self-supervised representation learning.

Lecture slides: https://drive.google.com/file/d/1JbP-02d6e8Hv3a8Fbaq9IYBRmJJPzygH/view?usp=sharing

The talk is based on our recent papers:

(1) ICLR 2021 paper 'Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study’,
Project page: http://zhiqiangshen.com/projects/LS_and_KD/index.html

(2) AAAI 2019 paper ‘MEAL: Multi-Model Ensemble via Adversarial Learning’,
Github: https://github.com/AaronHeee/MEAL

(3) NeurIPS 2020 workshop paper: ‘Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks’,
Github: https://github.com/szq0214/MEAL-V2

(4) CVPR 2021 paper ‘S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration’,
Github: https://github.com/szq0214/S2-BNN

Lecture abstract:

Deep neural networks have recently been recognized as one of the most successful techniques for a variety of learning tasks. In addition, it is obvious that the world will be populated with intelligent devices in the not-too-distant future, which will necessitate the execution of deep models on these low-cost, low-power and inexpensive hardware platforms. In this context, efficient deep learning has emerged as a critical concept and research target, with the goal of learning compressed deep models and developing training algorithms to increase the efficiency of model representations, data and label usage, and so on. Architecture search, distillation, pruning, binarization, quantization, and their learning methods for lowering the number of parameters and processing needs of deep models are some of the ways to get compressed networks and improved representation. In this talk, I'll cover one common approach: knowledge distillation for efficient deep learning, based on our previous several studies on understanding soft label in knowledge distillation, efficient knowledge distillation design, and its wide range of applications in visual recognition, self-supervised representation learning, etc.

Presenter Bio:

Zhiqiang Shen is a Postdoctoral Researcher at Carnegie Mellon University and Mohamed bin Zayed University of Artificial Intelligence, working with Prof Marios Savvides and Prof Eric Xing. He was a joint-training PhD student at Fudan University and University of Illinois at Urbana-Champaign. Starting from early 2022, he will join Hong Kong University of Science and Technology as a Research Assistant Professor in the Department of Computer Science and Engineering (CSE). His research interests span broad areas of efficient deep learning, machine learning, computer vision, etc. He has published 30+ top-tier papers on TPAMI, IJCV, ICML, ICLR, CVPR, ICCV, ECCV, AAAI, etc. with 2600+ google scholar citations.
More information about Zhiqiang can be found at http://zhiqiangshen.com/

-------------------------
Find us at:

All lectures are uploaded to our Youtube channel ➜ https://www.youtube.com/channel/UCHObHaxTXKFyI_EI8HiQ5xw

Newsletter for updates about more events ➜ http://eepurl.com/gJ1t-D

Sub-reddit for discussions ➜ https://www.reddit.com/r/2D3DAI/

Discord server for, well, discord ➜ https://discord.gg/MZuWSjF

Blog ➜ https://2d3d.ai

Peter Naf

2d3d.ai

Technology

3D Animation

3D Artists

Artificial Intelligence

Data Science

Reddit

Digital Signal Processing

Python

Education & Technology

3D Modeling

2D & 3D Modeling

Machine Learning

Image Processing

3D Printing

Deep Learning

Knowledge Distillation, Model Ensemble and Its Application on Visual Recognition

Online event

Share

2d3d.ai

Knowledge Distillation, Model Ensemble and Its Application on Visual Recognition

2d3d.ai

Details

Members are also interested in