【Academic Seminar】Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization
- Title: Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Acceleration
- Speaker: Prof. Xuefeng GAO, CUHK
- Time and Date: 2:00 pm – 3:00 pm, Thursday, October 11, 2018
- Venue: Boardroom, Dao Yuan Building
Abstract:
Stochastic gradient Hamiltonian Monte Carlo (SGHMC) is a variant of stochastic gradient with momentum where a controlled and properly scaled Gaussian noise is added to the stochastic gradients to steer the iterates towards a global minimum. Many works reported its empirical success in practice for solving stochastic non-convex optimization problems, in particular it has been observed to outperform overdamped Langevin Monte Carlo-based methods such as stochastic gradient Langevin dynamics (SGLD) in many applications. Although asymptotic global convergence properties of SGHMC are well known, its finite-time performance is not well-understood.
In this work, we provide finite-time performance bounds for the global convergence of SGHMC for solving stochastic non-convex optimization problems with explicit constants. Our results lead to non-asymptotic guarantees for both population and empirical risk minimization problems. For a fixed target accuracy level $\varepsilon$, on a class of non-convex problems, we obtain iteration complexity bounds for SGHMC that can be tighter than those for SGLD up to a square root factor. These results show that acceleration with momentum is possible in the context of non-convex optimization algorithms. This is a joint work with Mert Gurbuzbalaban and Lingjiong Zhu.
-----------------------------
Biography of speaker:
Xuefeng Gao is an assistant professor in the department of Systems Engineering and Engineering at Chinese University of Hong Kong. He received his Ph.D. degree in operations research from the H. Milton Stewart School of Industrial and Systems Engineering at Georgia Institute of Technology in 2013. He also holds a B.S. degree in mathematics from Peking University. His current research interests focus on applied probability, stochastic processes, data science and algorithmic trading.
