Peter L. Bartlett
[intermediate/advanced] Deep Learning: A Statistical Viewpoint
This course reviews tools for the analysis of the statistical performance of deep neural networks on classification and regression problems. It starts with uniform convergence properties, which show how this performance depends on notions of complexity, such as Rademacher averages, covering numbers, and combinatorial dimensions, and how these quantities can be bounded for neural networks. It also reviews the analysis of the performance of nonparametric estimation methods such as nearest-neighbor rules and kernel smoothing. Deep networks raise some novel challenges: despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We survey recent theoretical progress on this phenomenon of benign overfitting, focusing on regression problems with quadratic loss. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.
Uniform laws of large numbers
Statistical complexity of neural networks
Bias-variance decompositions and local methods
Benign overfitting: generalization with interpolating prediction rules
Peter L Bartlett, Andrea Montanari and Alexander Rakhlin (2021). Deep learning: A statistical viewpoint. Acta Numerica, 30, 87-201. doi:10.1017/S0962492921000027. Also arxiv:2103.09177 (and its references).
Probability theory, linear algebra.
Peter Bartlett is professor of Computer Science and Statistics at the University of California at Berkeley, Associate Director of the Simons Institute for the Theory of Computing, and Director of the Foundations of Data Science Institute. He works with Google Brain and has previously held positions at the Queensland University of Technology, the Australian National University and the University of Queensland. His research interests include machine learning and statistical learning theory, and he is the co-author of the book Neural Network Learning: Theoretical Foundations. He has been Institute of Mathematical Statistics Medallion Lecturer, NeurIPS Posner Lecturer, winner of the Malcolm McIntosh Prize for Physical Scientist of the Year, and Australian Laureate Fellow, and he is a Fellow of the IMS, Fellow of the ACM, and Fellow of the Australian Academy of Science.