Generative AI for 3D Content

Abstract:

While generative AI has achieved remarkable success in creating language, images, and videos, its application in 3D content, which is the heart of several domains, such as VR/AR, film, gaming, and metaverse, encounters fundamental challenges due to the scarcity of 3D training data and increased complexities inherent in 3D. In this talk, I will present my research on developing 3D generative AI models to create realistic, high-quality, and diverse 3D content by leveraging the domain knowledge from computer graphics. First, I will discuss how incorporating the 3D modeling techniques from computer graphics could not only enhance efficiency and unlock new capabilities in generating relightable and simulable 3D content, but also regularizes the generation behavior, enforcing the model to focus specifically on the 3D geometry. Second, I will show how leveraging 2D foundation models could facilitate high-quality and diverse 3D content generation for geometry, appearance, and semantics by combining our 3D modeling techniques with differentiable rendering. The techniques we build are able to turbocharge applications ranging from 3D generation from a single image, 3D reconstruction from multi-view images, 3D generative modeling, and text to 3D generation.  Finally, I will discuss the future direction of generating realistic 3D virtual worlds to enable immersive interactions with humans.

Bio:

Jun Gao is a PhD candidate at the University of Toronto, advised by Prof. Sanja Fidler. He is also a Research Scientist at NVIDIA. His research lies in the interaction of computer vision, computer graphics, and machine learning, focusing on developing generative AI models to create 3D content for reconstructing, generating, and simulating lifelike 3D worlds. His research received the SIGGRAPH Asia Best Paper award. Previously, Jun received his Bachelor’s degree from Peking University.