Leveraging Priors of Generative Models

Hybrid Event: 3725 BBB / Zoom

Abstract: Image and video generative models have deep knowledge of the world, but text-prompting—the primary way in which we interact with them—only scratches the surface of this understanding. In this talk, we will discuss ways to adapt these generative models to leverage and expose their powerful priors. In particular, we show that generative image models can create optical illusions zero-shot, subverting the way that humans perceive images and by doing so, indicating that these models may have an understanding of human visual perception. We then show how adding motion conditioning to a generative video model not only enables more natural user control, but also exposes the underlying physics and causality that the model has learned in a compelling way.