Joachim M. Buhmann
[introductory/advanced] Model and Algorithm Validation for Data Science
Data are everywhere these days and they amount to an unimaginable complexity. MRF imagery, digital biopsies, Omics data in personalized medicine, the microwave background radiation of the universe, social network traces, the financial flow around the globe, etc., digitalization has enables us humans to perceive facets of reality in an unprecedented complexity. Modern computing machinery and i.e., high performance algorithmics enables us to build model that are too complex to be memorized or even understood but we can control such models and process their predictions. An automated scientific method with algorithms in experimental design, analysis and theory building arises on the methodological horizon and it will transform empirical sciences with many heterogeneous degrees of freedom from descriptive enterprises to predictive inferences. The foremost challenge will confront us with validating these algorithms, especially determining their statistical correctness. This machine learning disruption will also transform computer science in a fundamental way.
The course is based on the assumption that data science algorithms sample from posterior probability distributions of hypotheses given empirical data. These posteriors have to be validated in the sense that broad posteriors are uninformative and highly peaked posteriors are unstable to data fluctuations. The tradeoff between the underfitting and the overfitting extremes require information theoretic considerations that exploit the minimum description length idea in combination with rate distortion theory.
- Maximum entropy inference and Gibss distributions
- Minimum description length principle
- AIC, BIC, stability selection
- Tishby’s Information Bottleneck Method
- Information Theoretic Model Validation, algorithms as time evolving posterior distributions
- Examples in approximate sorting, approximate spanning trees
- Pipeline tuning in biomedical applications
V. Wegmayr and J. M. Buhmann, “Entrack: Probabilistic spherical regression with entropy regularization for fiber tractography,” International Journal of Computer Vision, vol. 129, no. 3, pp. 656–680, 2021.
J. M. Buhmann, J. Dumazert, A. Gronskiy, andW. Szpankowski, “Posterior agreement for large parameter-rich optimization problems,” Theoretical Computer Science, vol. 745, pp. 1–22, 2018.
J. Buhmann, A. Gronskiy, M. Mihalák, T. Pröger, R. Srámek, and P. Widmayer, “Robust optimization in the presence of uncertainty: A generic approach,” Journal of Computer and System Sciences, vol. 94, pp. 135–166, 2018.
N. Gorbach, M. Tittgemeyer, and J. M. Buhmann, “Pipeline validation for connectivity-based cortex parcellation,” NeuroImage, vol. 181, pp. 219–234, 2018.
Thomas Cover and Jay Thomas, Elements of information theory. Wiley, 2006.
N. Tishby, F. Pereira, W. Bialek, “The Information Bottleneck Method”. The 37th Allerton Conference on Communication, Control, and Computing. pp. 368–377, 1999.
Peter D. Grünwald, “The Minimum Description Length Principle”. MIT Press, 2007.
Shu-Cherng Fang, Jay R. Rajasekera, H.-S. Jacob Tsao, Entropy Optimization and Mathematical Programming. Kluwer Academic Publishers, 1997.
Basic knowledge of statistics and probability theory, familiarity with statistical learning theory concepts.
Joachim M. Buhmann is a Professor for Computer Science at ETH Zurich since October 2003. He heads the Institute for Machine Learning at the Department of Computer Science. He studied physics at the Technical University of Munich and was awarded a PhD for his work on artificial neuronal networks. After research appointments at the University of Southern California, Los Angeles, and at the Lawrence Livermore National Laboratory he joined the University of Bonn as professor for practical computer science (1992 – 2003).
Buhmann’s research interests cover theory and applications of machine learning and artificial intelligence, as well as a wide range of subjects related to information processing in the life sciences. His conceptual and theoretical work on machine learning investigates the central question, how complex models and algorithms in data analysis (Big Data) can be validated, if they are estimated from empirical observations.
Joachim Buhmann served as Director of Studies for Computer Science (2008 – 2013) and as Vice-Rector for Study Programmes (2014-2017) of ETH Zurich. Since 2017, he represents the field of Data Science in the Swiss National Science Foundation (SNSF) . The German Pattern Recognition Society (DAGM) awarded him an honorary membership in 2017. He was elected as an individual member of the Swiss Academy of Engineering Sciences (SATW) in the same year. In 2020, he was appointed as a fellow of the International Association for Pattern Recognition (IAPR).