Lior Yariv- Universal Differentiable Renderer for Implicit Neural Representation
Details
Collaboration with women in AI -
Learning 3D shapes with 2D supervision (i.e., images) is a fundamental computer vision problem. A recent successful neural networks approach to solve this problem is to use a differentiable rendering system coupled with a choice of 3D geometry representation. Differential rendering systems are mostly based on ray casting/tracing, or rasterization, while popular models to represent 3D geometry include point clouds, triangle meshes, implicit representations defined over volumetric grids, and recently also neural implicit representations, namely, zero level sets of neural networks. Neural implicit representations have become a popular choice, mainly due to their flexibility to represent surfaces with arbitrary shape and topology, and produce smooth surface approximation without a fixed discretization.
However, so far differentiable rendering systems with implicit neural representations did not incorporate lighting and reflectance properties required for producing faithful appearance of 3D geometry in images. The main challenge seems to be relating, in a differentiable way properties of the implicit surface and the parameters of the neural network representing it.
In this talk I will introduce Universal Differentiable Renderer (UDR), a neural network architecture for learning 3D shapes with 2D supervision that can provably approximate reflected light from an implicit neural representation of a 3D surface.
Link to the paper: https://arxiv.org/pdf/2003.09852.pdf