Mathematics of Machine Learning 2024
This is the main website for the Mathematics of Machine Learning course in the spring of 2024, as part of the bachelor of mathematics at the University of Amsterdam. Visit this page regularly for changes and updates.
| Instructor: | Tim van Erven | (tim@ No spam, please timvanerven. No really, no spam nl) | |
| Teaching Assistants: | Bharti Bharti | (b.bharti@ No spam, please uva. No really, no spam nl) | Group A |
| Gerrit Oomens | (g.oomens@ No spam, please uva. No really, no spam nl) | Group B |
General Information
Machine learning is one of the fastest growing areas of science, with far-reaching applications. This course gives an overview of the main techniques and algorithms. The lectures introduce the definitions and main characteristics of machine learning algorithms from a coherent mathematical perspective. In the workgroups, students will both solve mathematical exercises to deepen their understanding, and apply algorithms from the course to a selection of data sets using Python Jupyter notebooks.
We will use Canvas for announcements, grades and submitting homework. I will also post my handwritten lecture notes there.
Required Prior Knowledge
- Linear algebra, gradients, convexity
- Ability to write mathematical proofs
- Programming in Python with Jupyter notebooks
- Writing in LaTeX
Although mainly targeting mathematics students, the course is accessible to other science students (AI, CS, physics, …) with an interest in mathematical foundations of machine learning.
Lectures and Exercise Sessions
- Weekly lectures:
- weeks 7-12 from 11h00-13h00 on Tuesdays:
- in room SP G1.18 in weeks 7, 8.
- in room SP C1.112 in weeks 9-12.
- weeks 15-20 except April 29 from 11h00-13h00 on Mondays:
- in room SP F1.02 in weeks 15-19.
- in room SP D1.114 in week 20.
- weeks 7-12 from 11h00-13h00 on Tuesdays:
- Weekly exercise classes from 15h00-17h00 on Thursdays, starting in the
second week of the course:
- Group A: in room SP D1.112 in weeks 8-12, 14-17, 20
- Group B: in room SP G5.29 in weeks 8-12; in room SP G2.02 in weeks 14-17, 20
Examination Form
The course grade consists of the following components:
- Homework assignments. H = Average of homework grades,
excluding the lowest homework grade. - Two exams: midterm (M) and final (F).
The final grade is computed as 0.3H + 0.3M + 0.4F. If between 5 and 6, it is rounded to a whole point: 5 or 6. Otherwise it is rounded to half points.
Exams (closed book):
- Midterm: March 28, 9h00-11h00 in room SP F1.02
- Final exam: May 29, 18h00-21h00 in room SP C0.05
- Resit exam: July 2, 9h00-12h00 in room SP B0.206
The midterm will be about the first half of the course. The final exam will only be about the second half of the course. The resit exam (R) will cover both halves; it will replace both the midterm and the final exam, with final grade 0.3H + 0.7R. Both exams will be closed book, meaning that it is not allowed to use external resources during the exam.
Course Materials
The main book for the course is The Elements of Statistical Learning (ESL), 2nd edition, by Hastie, Tibshirani and Friedman, Springer-Verlag 2009. In addition, we will use selected parts from Ch. 18 of Computer Age Statistical Inference: Algorithms, Evidence and Data Science (CASI) by Efron and Hastie, Cambridge University Press, 2016. Some supplementary material will also be provided, as listed in the Course Schedule.
Both books are freely available online, but you may consider buying a paper copy of the ESL book, because you will need to study many of its chapters. The standard edition of ESL is hard cover, but there also exists a cheaper soft-cover edition for €39.99. To get the cheaper offer, open this link from inside the university network.
Course Schedule
This schedule is subject to change during the course. Literature marked ‘optional’ is recommended for background, but will not be tested on the exam. TBA=To Be Announced.
| Date | Topics | Literature |
|---|---|---|
| Feb. 13 | Supervised learning intro: classification and regression (overfitting 1), linear regression for classification (overfitting 2), nearest neighbor classification (overfitting 3). |
Ch. 1.
Sect. 2.1, 2.2, 2.3. |
| Feb. 20 |
Curse of dimensionality.
Statistical decision theory: expected prediction error (overfitting 4), Bayes-optimal prediction rule. Empirical Risk Minimization. Interpretation of least squares as ERM. |
Sect. 2.4, 2.5. |
| Feb. 27 |
Cross-validation.
Model selection for regression I: best-subset selection. |
Sect. 7.10.1, 7.10.2; optionally: 7.12.
Sect. 3.1, 3.2 up to 3.2.1, 3.3. |
| Mar. 5 | Model selection for regression II: ridge regression and lasso. | Sect. 3.4 up to 3.4.3. |
| Mar. 12 |
Surrogate losses.
Logistic regression. |
Sect. 4.4 intro before 4.4.1 + definition of regularization at start of sect. 4.4.4. |
| Mar. 19 |
Decision trees for classification and regression,
and random forests.
Bagging. Q&A session. |
Sect. 9.2,
Sect. 8.7. |
| Mar. 28 | No Lecture, Midterm Exam on Thursday | |
| Apr. 1 | Easter, no lecture | |
| Apr. 8 | Boosting (AdaBoost), boosting as forward stagewise additive modeling. | Sect. 10.1, 10.2, 10.3., 10.4, 10.5, 10.6 (in 10.6 only the part about classification). |
| Apr. 15 | SVMs I: Optimal separating hyperplane, support vector machine (SVM), SVM learning as regularized hinge loss fitting. | Sect. 4.5.2, 12.2, 12.3.2. |
| Apr. 22 | SVMs II: dual formulation, kernel trick. | Sect. 12.3.1. Optionally: Ch. 5 from Boyd and Vandenberghe book |
| Apr. 29 | Lecture-free Week | |
| May 6 |
Unsupervised learning: K-means clustering.
Stochastic Optimization. |
Sect. 14.3 before 14.3.1; Sect. 14.3.6. NB. The book gives the wrong definition for K-means in Sect. 14.3.6, see erratum.
Handout about stochastic optimization. |
| May 13 |
Neural networks/deep learning:
gradient descent with backpropagation,
convolutional layers.
Q&A session. |
From Ch. 18 of the CASI book: chapter intro, Sect. 18.1, Sect. 18.2 (except accelerated gradient methods), 18.4. |
| May 20 | Pentecost, no lecture | |
| May 29 | Final Exam |
Homework Assignments
The homework assignments will be made available here. It is allowed to work together in pairs of two students, which can change per assignment. It is not allowed to collaborate with other people. In case you miss a deadline because of illness or other special circumstances, contact Tim to discuss possible solutions.
Submit via Canvas. Write your answers in LaTeX.
| Homework | Extra Files | Available | Deadline |
|---|---|---|---|
| Homework 1 | Homework1-start.ipynb | 22 Feb | 28 Feb, 13h00 |
| Homework 2 | 28 Feb | 6 Mar, 13h00 | |
| Homework 3 | Homework3-start.ipynb | 4 Mar | 13 Mar, 13h00 |
| Homework 4 | Homework4-start.ipynb | 13 Mar | not graded |
| Homework 5 | - | 10 Apr | 17 Apr, 13h00 |
| Homework 6 | - | 17 Apr | 24 Apr, 13h00 |
Further Reading
Here is a list of references for advanced further reading. These are all optional, and will not be tested on the exam.
- Machine Learning Theory: I recommend the free book by Shalev-Shwartz and Ben-David, which we also use in the MasterMath course Machine Learning Theory.
- Convex optimization: the free book by Boyd and Vandenberghe provides a very nice introduction. For a more extensive overview, see the free book by Bubeck.
- Deep learning: if you want to get up to date on the practice of deep learning, I recommend the Dive into Deep Learning interactive online book.