Improving Human Pose Estimation with Generative Models and Physics Simulation

Ye Yuan, Research Scientist, NVIDIA Research

Abstract:
Human pose estimation is essential for many vision applications in mixed reality, autonomous driving, and robotics. In this talk, I will show how generative models and physics simulation can be leveraged to improve aspects of human pose estimation where traditional kinematic methods fall short. First, human pose estimation often suffers from occlusion and ambiguity caused by moving cameras. To tackle this, we leverage motion generation models to perceive invisible human motions from videos and constrain the joint reconstruction of human movement and camera motion. Second, the laws of physics are often not observed by kinematic pose estimation methods, which leads to physically-implausible pose estimates with pronounced artifacts such as jitter, foot sliding, and penetration. To address this, I will present an RL-based character control framework to model human motions in physics simulation. It allows us to significantly improve physical plausibility for third-person human pose estimation as well as the challenging egocentric pose estimation using wearable cameras.

Bio:
Ye Yuan is a research scientist at NVIDIA Research. He received his Ph.D. in Robotics from Carnegie Mellon University (CMU) in 2022, where he worked with Prof. Kris Kitani. His research lies at the intersection of computer vision, robotics, and machine learning. He is particularly interested in simulation, reinforcement learning, 3D computer vision, generative models, embodied agents, and digital humans. He is a recipient of the Qualcomm Innovation Fellowship and the NVIDIA Graduate Fellowship. He has collaborated with Disney Research and Facebook Reality Lab through internships. He received his M.S. in computer science from CMU and B.E. in computer science from Zhejiang University.
https://ye-yuan.com

Join Zoom Meeting
https://us02web.zoom.us/j/82336841136?pwd=Q1d6WFhJMzRYR2hXcjREZlhRVE15Zz09
Meeting ID: 823 3684 1136
Passcode: 376961
One tap mobile
+16694449171,,82336841136# US
+16699006833,,82336841136# US (San Jose)
Find your local number: https://us02web.zoom.us/u/kd0KAUPAGb

YouTube
TBA