Analysis of Policy Gradient Descent for Control: Global Optimality via Convex Parameterization

ABSTRACT: Motivated by the widely used “policy gradients” and related methods in reinforcement learning, revisiting these methods in the context of classical control problems has been a recent focus of study. We start by examining the convergence and optimality of these methods for the infinite-horizon Linear Quadratic Regulator (LQR), where we show that despite nonconvexity (with respect to policy parameters), gradient descent converges to the optimal policy under mild assumptions. Next, we make a connection to convex parameterization techniques, such as Youla parameterization—which leads to a unified way to prove similar results for a whole host of control design problems, as long as they admit a convex parameterization.

BIO: Maryam Fazel is the Moorthy Family Professor of Electrical and Computer Engineering at the University of Washington, with adjunct appointments in Computer Science and Engineering, Mathematics, and Statistics. Maryam received her MS and PhD from Stanford University, and her BS from Sharif University of Technology in Iran, and was a postdoctoral scholar at Caltech before joining UW. She is a recipient of the NSF Career Award, UWEE Outstanding Teaching Award, and a UAI conference Best Student Paper Award with her student. She directs the Institute for Foundations of Data Science (IFDS), a multi-site, collaborative NSF TRIPODS Institute. Her current research interests are in the area of optimization in machine learning and control.

***Event will take place via Zoom. Zoom link and password will be distributed to the Controls Group e-mail list-serv. To join this list-serv, please send an (empty) email message to [email protected] with the word “subscribe” in the subject line. Zoom information is also available upon request to Katherine Godwin ([email protected]).