Introduction

Introduction#

\[ \newcommand{\bm}[1]{\boldsymbol{#1}} \newcommand{\Det}[1]{|\boldsymbol{#1}|} \newcommand{\bigO}{\mathcal{O}} \newcommand{\var}{\mathrm{Var}} \newcommand{\cov}{\mathrm{Cov}} \newcommand{\Prob}{\mathrm{Prob}} \newcommand{\mean}[1]{\langle #1 \rangle} \]

During the last two decades there has been a swift and remarkable development of machine learning techniques and algorithms, and their impact is felt not only in science and technology but also in the humanities, the social sciences, medicine and law – indeed in almost every discipline one can name. The applications are innumerable, from self-driving cars to the solution of high-dimensional differential equations or complicated quantum mechanical many-body problems. Machine learning is perceived by many as one of the main disruptive technologies of our time.

Statistics, data science and machine learning form central fields of research in modern science. They describe how to learn and make predictions from data, and they allow us to extract correlations about physical processes and the underlying laws of motion from large data sets. Such large data sets appear frequently in essentially all disciplines, from the traditional science, technology, mathematics and engineering fields to life science, law, educational research, the humanities and the social sciences. It has become more and more common to see research projects on big data in, for example, the social sciences, where extracting patterns from complicated survey data is one of many research directions. A solid grasp of data analysis and machine learning is thus becoming central to scientific computing in many fields, and competences within machine learning and scientific computing are strongly requested by many potential employers. The latter can hardly be overstated: familiarity with machine learning has almost become a prerequisite for many of the most interesting employment opportunities, whether in bioinformatics, life science, physics or finance, in the private or the public sector. This author has had several students who were hired on the basis of their skills in scientific computing and data science, often with only a marginal knowledge of machine learning itself.

Machine learning is a subfield of computer science, and is closely related to computational statistics. It evolved from the study of pattern recognition in artificial intelligence research, and has made contributions to tasks like computer vision, natural language processing and speech recognition. Many of the methods we study here are, however, just as strongly rooted in basic mathematics and physics. A restricted Boltzmann machine is a spin system in thermal equilibrium; a diffusion model is a discretised stochastic differential equation; a convolutional layer is a discrete convolution, with all the Fourier theory that goes with it; and the algorithm that trains all of them is the chain rule. This is not a coincidence, and the reader who comes to the subject from the physical sciences already possesses a good deal of what is needed.

This introductory chapter says what machine learning is taken to mean here, what this book aims to do, how it is put together, and which ethical obligations come with the material. The last of these is not an ornament. We end the chapter with it deliberately.

Machine learning, a small and probably biased introduction#

Ideally, machine learning represents the science of giving computers the ability to learn without being explicitly programmed. The idea is that there exist generic algorithms which can find patterns in a broad class of data sets without our having to write code specifically for each problem; the algorithm builds its own logic from the data. One should however always keep in mind that machines and algorithms are to a large extent developed by humans, and that the insight and knowledge we have about a specific system play a central role when we develop a specific machine learning algorithm. This book is written from that conviction. Almost every chapter contains a place where knowing something about the problem – that an image is translation invariant, that a probability must be positive, that a differential equation must satisfy a boundary condition – buys more than a larger model would.

Machine learning is an extremely rich field, in spite of its young age. The increase in computational capability over the last decades has been followed by the development of methods for analysing and handling large data sets, relying heavily on statistics, computer science and mathematics. Popular software libraries written in Python, such as Scikit-learn, TensorFlow, PyTorch and Keras, all freely available at their respective GitHub sites, encompass communities of developers numbering in the thousands or more, and the number of contributors keeps increasing. Not all the algorithms and methods can be given a rigorous mathematical justification – decision trees and random forests are good examples – which opens up large rooms for experimentation, trial and error, and thereby for exciting new developments. A solid command of linear algebra, multivariate calculus, probability theory, statistical data analysis, error estimation and Monte Carlo methods is nevertheless central to a proper understanding of most of the algorithms we shall discuss, and Chapters 1 and 2 are there to supply it.

The approaches to machine learning are many, but are often split into two main categories. In supervised learning we know the answer to a problem and let the computer deduce the logic behind it. In unsupervised learning we look for patterns and relationships in data sets without any prior knowledge of the system. Some authors operate with a third category, reinforcement learning, a paradigm inspired by behavioural psychology in which learning is achieved by trial and error, solely from rewards and punishments. The first two categories organise this book; the third is discussed, but not developed, in Chapter 18.

Another way to categorise machine learning tasks is by the desired output of the system. Three of the most common tasks are the following.

Classification. Outputs are divided into two or more classes, and the goal is to produce a model that assigns inputs to one of these classes. Identifying digits from pictures of handwritten ones is the standard example. Classification is usually supervised learning, and it is the subject of Chapters 5, 6 and 7, and again of Chapter 8 onwards.
Regression. Finding a functional relationship between an input data set and a reference data set: the goal is to construct a map from inputs to continuous output values. This is where we begin, in Chapter 3.
Clustering. Data are divided into groups sharing certain traits, without knowing the groups beforehand. It is thus a form of unsupervised learning, and it is closely related to the dimensionality reduction we meet as principal component analysis in Section Principal component analysis and again, in a nonlinear guise, in Chapter 12.

The methods we cover have three ingredients in common, irrespective of whether we deal with supervised or unsupervised learning.

The first ingredient is our data set, which is normally subdivided into training, validation and test data. Many find the most difficult part of applying machine learning to be the setting up of the data in a meaningful way. Why the subdivision is necessary at all, and what goes wrong when it is neglected, is the subject of Sections Training error, test error and generalisation and Cross-validation.
The second is a model, which is normally a function of a set of parameters. The model reflects our knowledge of the system, or our lack of it. If we know that our data behave roughly like a polynomial, fitting a polynomial of some degree determines our model.
The third is a cost function, also called a loss function or an error function, which measures how well the model reproduces the data it is trained on.

The third ingredient deserves a warning already here, because it is one of the threads that runs through the entire book. The cost function is not a free design decision. Specify what the target is assumed to be, and the cost function follows: Gaussian noise gives the squared error of Section Ordinary least squares, Bernoulli observations give the cross entropy of Section Maximum likelihood and the cross-entropy, a categorical target gives the multiclass cross entropy that pairs with the softmax of Section More than two classes: the softmax, and even the penalties are not free – Section A Bayesian reading shows that a flat prior gives ordinary least squares, a Gaussian prior gives Ridge regression and a Laplace prior gives the Lasso. A reader who acquires the habit of asking which assumption generated a given loss will find that the strange-looking objectives of the generative chapters are not strange at all.

At the heart of essentially all machine learning algorithms we find a minimisation or optimisation problem, and a large family of methods for solving it goes under the name of gradient methods. These are developed in Chapter 4 and then used, without further apology, everywhere else.

A remark on what is new and what is not

Of the ingredients above, only the third is peculiar to machine learning, and even that only mildly. Fitting a model by minimising a cost function is what a physicist does when fitting a resonance curve, and least squares is two centuries old. What has changed is the size of the model class, the fact that the features are learned rather than chosen, and the availability of a cheap and exact gradient for arbitrarily complicated compositions of functions. That last point – reverse-mode automatic differentiation, Section Automatic differentiation – is the technical development that made the rest possible.

A frequentist approach to data analysis#

When you hear phrases like predictions and estimations or correlations and causations, what do you think of? Perhaps of the difference between classifying new data points and generating new ones. Or perhaps you consider that correlations are symmetric statements – if \(A\) is correlated with \(B\), then \(B\) is correlated with \(A\) – while causation is directional: if \(A\) causes \(B\), \(B\) need not cause \(A\).

These concepts mark, in some sense, the difference between machine learning and statistics. In machine learning and prediction-based tasks we are often interested in algorithms that learn patterns from given data in an automated fashion, and then use these learned patterns to make predictions about new data. In many cases our primary concern is the quality of the predictions, and we are less concerned about the underlying patterns that were learned in order to make them.

In machine learning we normally use a so-called frequentist approach, where the aim is to make predictions and find correlations. We focus less on extracting a probability distribution function, which could in turn be used to make estimations and find causations, such as: given \(A\), what is the likelihood of finding \(B\)?

The distinction is one of emphasis rather than of principle, and this book crosses it in both directions. Chapter 2 is written from the frequentist side: an estimator is a random variable, and what we want to know about it is its bias and its variance. Section A Bayesian reading then reads exactly the same regression estimators as maximum a posteriori estimates under different priors, and the whole of Part IV is Bayesian in spirit, since a generative model is a statement about \(p(\bm{x})\) and nothing else. The reader who keeps both readings available will find several results easier, not harder: the Ridge estimator is a shrunken least-squares estimator and a Gaussian posterior mode, and it is useful to be able to see it as both.

What is a good model?#

In science and engineering we often end up in situations where we want to infer, or learn, a quantitative model \(M\) for a given set of sample points \(\bm{X} \in [x_1, x_2,\dots ,x_N]\). As we shall see repeatedly, we could try to fit these data points to a straight line, or, if we wish to be more sophisticated, to a more complex function.

The reason for inferring such a model is that it serves many useful purposes. On the one hand, the model can reveal information encoded in the data, or underlying mechanisms from which the data were generated; we could discover important correlations that admit an interesting physical interpretation. In addition, a model simplifies the representation of a given data set and helps us make predictions about future samples.

A first important consideration to keep in mind is that inferring the correct model for a given data set is an elusive, if not impossible, task. The fundamental difficulty is that if we are not specific about what we mean by a correct model, there could easily be many different models that fit the given data set equally well.

The central question is then: what leads us to say that a model is correct, or optimal, for a given data set? To make the model inference problem well posed, that is, to guarantee that there is a unique optimal model for the given data, we need to impose additional assumptions or restrictions on the class of models considered. We should not be looking for just any model that can describe the data; we should look for a model \(M\) that is the best among a restricted class of models. In addition, to make the problem computationally tractable, we need to specify how restricted that class needs to be. A common strategy is to start with the simplest class of models that is just necessary to describe the data, or to solve the problem at hand. More precisely, the model class should be rich enough to contain at least one model that fits the data to the desired accuracy, and yet be restricted enough that it is relatively simple to find the best model within it.

The most popular strategy is therefore to start from the simplest class of models and to increase the complexity only when the simpler models become inadequate. If we work with a regression problem, one may first try linear models, followed by more complex ones.

How to evaluate which model fits the data best is something we come back to over and over again. The single most important tool is the bias-variance decomposition of Section The bias-variance tradeoff, which states that the expected test error splits into a bias term, a variance term and an irreducible noise floor, and that the first two move in opposite directions as model complexity grows. It is used in almost every chapter of this book, and it is the reason why the answer to “which model is best?” is never “the one with the smallest training error”. The two structurally different ways of exploiting the decomposition – constraining a single model, as in Section Ridge regression, and averaging many, as in Chapter 7 – are among the few ideas in this book that will still be useful when every architecture in it has been superseded.

Aims and learning outcomes#

This book aims at giving an overview of central aspects of statistical data analysis as well as of the central algorithms used in machine learning. Its origin is a one-semester course, and the pedagogical order of that course has been kept: nothing is used before it has been derived.

Hands-on work with data and algorithms plays a central role, and the hope is, through the exercises and the accompanying programs, to expose the reader to fundamental research problems in these fields, with the aim of reproducing state-of-the-art scientific results. The reader will learn to develop and structure codes for studying such systems, become acquainted with computing facilities, and learn to handle large scientific projects. A good scientific and ethical conduct is emphasised throughout. More specifically, the reader will

learn about basic data analysis, statistical inference, Monte Carlo methods, data optimisation and machine learning;
be capable of extending the acquired knowledge to other systems and cases;
gain an understanding of the central algorithms used in data analysis and machine learning;
acquire knowledge of central aspects of Monte Carlo methods, Markov chains and Gibbs samplers, and of their possible applications, from numerical integration to the simulation of physical systems;
understand methods for regression and classification;
learn about neural networks of several architectures, Boltzmann machines and modern generative models;
work on numerical projects that illustrate the theory. The projects play a central role, and the reader is expected to know a modern programming language such as Python or C++, in addition to having a basic knowledge of linear algebra, typically taught during the first one or two years of undergraduate studies.

The topics covered span statistical data analysis and its basic concepts – expectation values, variance, covariance, correlation functions and errors – via well-known probability distributions such as the uniform, binomial and Poisson distributions and the simple and multivariate normal distributions, to central elements of Bayesian statistics and modelling. We also remind the reader of central elements from linear algebra and of the standard methods built upon it: the family of gradient descent methods, the singular value decomposition, and least-squares methods for parameterising data. We cover Monte Carlo methods, Markov chains, and well-known algorithms for sampling stochastic events such as the Metropolis-Hastings and Gibbs sampling methods. An important aspect of all our calculations is a proper estimation of errors, and we therefore discuss resampling techniques such as blocking, the bootstrap and the jackknife, together with the infamous bias-variance trade-off. The remainder of the material covers the algorithms of machine learning proper.

The structure of this book#

The book is organised in five parts and a closing chapter. It is not a catalogue of methods. A catalogue would have been easier to write and would have dated faster. What the book tries to do instead is to derive a small number of ideas carefully and then follow them until they stop working, because the place where an idea stops working is the place where the next one is invented. Read in that order, the seventeen chapters are not seventeen subjects but one subject seventeen times, and Chapter 18 closes the book by making that claim explicitly and drawing the whole as a graph.

The table below is the short version; the subsections that follow are the long one.

Ch.	Title	What it adds
Part I – Mathematical and statistical basis
1	Linear algebra	The objects: design matrices, projections, matrix calculus, the singular value decomposition and the condition number.
2	Probability and statistics	The discipline: estimators as random variables, the bias-variance decomposition, resampling, Markov chains.
Part II – Classical methods
3	Linear regression	Least squares derived three times, Ridge, the Lasso, kernels.
4	Optimisation	What to do when the closed form is withdrawn: gradient descent and its adaptive descendants.
5	Logistic regression	One function changes, and the loss changes with it.
6	Support vector machines	A third loss, reached from geometry, belonging to the same family.
7	Trees and ensembles	Improving a model by averaging copies of it rather than by changing it.
Part III – Deep learning
8	Neural networks	The hinge of the book: a hidden layer, and backpropagation.
9	Differential equations	The same network, asked a different question.
10	Convolutional networks	Weights that are local and shared; equivariance under translation.
11	Recurrent networks	Weights shared across time, and why long memories are hard.
12	Autoencoders	A bottleneck, and its exact identity with principal components.
13	Transformers	Weights computed from the data; equivariance under all permutations.
Part IV – Generative methods
14	Boltzmann machines	Refuse to compute the normalisation, and sample instead.
15	Variational autoencoders	Bound it.
16	Diffusion and flows	Learn a score, or restrict the map.
17	Adversarial networks	Abandon the likelihood and train a classifier in its place.
Part V and closing
18	Concluding remarks	The threads, a working protocol, the omissions, and where the methods are being taken in the sciences.

Part I: the mathematical and statistical basis#

The two opening chapters are not preliminaries in the usual sense of material one may skip.

Chapter 1 supplies the objects. A data set is a design matrix, a model is very often a matrix acting on a parameter vector, and fitting is projection onto a column space. The chapter develops the algebraic language – vectors and matrices, orthogonality and projection, derivatives with respect to vectors and matrices in Section Derivatives with respect to vectors and matrices, the spectral decomposition, the covariance matrix – and then asks how these objects are actually computed, from Gaussian elimination and the LU and Cholesky factorisations through the iterative and conjugate gradient methods of Sections Iterative methods for linear systems and The conjugate gradient method to the singular value decomposition of Section The singular value decomposition. It also supplies the one number, the condition number, that will reappear in three quite different roles: as the loss of precision in a linear solve, as the variance of a fitted parameter, and as the number of iterations gradient descent will need.

Chapter 2 supplies the discipline. Every quantity computed from a finite sample is itself a random variable, with a bias and a variance. The chapter builds the probability distributions we shall need, expectation values and moments, covariance and correlation, the central limit theorem, and then the statistics of estimators: the properties of the least-squares estimator in Section Statistical properties of the least-squares estimator, the distinction between training and test error in Section Training error, test error and generalisation, the bias-variance decomposition of Section The bias-variance tradeoff, the resampling methods of Section Resampling: the jackknife and the bootstrap and cross-validation in Section Cross-validation. It closes with random numbers, Markov chains and the Metropolis algorithm of Section Markov chains and the Metropolis algorithm, which is not needed again until Chapter 14 but is best learned here.

Without the first of these chapters the algebra of the later ones is unreadable; without the second, their numbers are unbelievable.

Part II: classical methods#

The five chapters that follow are, deliberately, largely the same model seen from different angles.

Chapter 3 derives the least-squares estimator three times – as an optimisation problem, as a projection, and as a maximum-likelihood estimate – because each derivation generalises differently and all three are needed later. It then adds a penalty and obtains Ridge regression and the Lasso, reads both as priors in Section A Bayesian reading, and lifts the whole construction into a feature space with the kernel methods of Section Kernel methods: regression without coordinates. Two extended examples run through the chapter: the Franke function of Section A complete example: the Franke function and the one-dimensional Ising model of Section A second example: the one-dimensional Ising model, the latter chosen because its exact answer is known and can be compared against.

Chapter 4 is what happens when the closed form is withdrawn. It begins with convexity and Newton’s method, works through plain and stochastic gradient descent, and then derives momentum, AdaGrad, RMSProp and Adam as answers to a single question – how should the step length adapt? – before admitting, in Section None of these can compete with Newton’s method, that none of them can compete with Newton’s method except in the one respect that matters at scale. Section Automatic differentiation explains why the gradient is cheap at all.

Chapter 5 changes one function, the identity to the sigmoid, and finds that the loss changes with it, without being chosen. It is the first classification chapter, it introduces the cross entropy and the softmax, and it introduces the measures – accuracy, precision, recall, the confusion matrix, the ROC curve – by which every classifier in the rest of the book is judged.

Chapter 6 arrives at a third loss from a purely geometric principle, the maximum margin, and finds that it belongs to the same family: Ridge regression, logistic regression and the support vector machine turn out to be one model with one penalty and three losses, which is the observation Section One penalty, three losses closes on. On the way it supplies the Lagrange duality and the Karush-Kuhn-Tucker conditions [boyd2004] that are useful well beyond this chapter.

Chapter 7 is the odd one out, and is the more valuable for it. It is the only place in the book where a model is improved not by changing it but by averaging many copies of it, and Section Why averaging helps gives the specification that every ensemble in the sciences obeys. Bagging, random forests, AdaBoost, gradient boosting and XGBoost then follow as variations on that one equation, with boosting reversing the strategy: instead of averaging strong learners to remove variance, it sums weak ones to remove bias.

Part III: deep learning#

Chapter 8 is the hinge of the book. It opens by admitting that inserting a hidden layer costs every guarantee the previous chapters relied on: no convexity, no closed form, no normal equations, no optimality conditions to check the answer against. What is left is gradient descent and the chain rule, and the rest of the book is a demonstration that this is enough. The chapter derives the four equations of backpropagation in Section The four equations of backpropagation, and they are then used, unmodified, by every architecture that follows. It is also careful, in Section The universal approximation theorem, to keep apart three questions that are routinely run together: whether the model class contains the function, whether gradient descent will find it, and whether the result will generalise. The universal approximation theorem [cybenko1989, hornik1991] answers only the first.

The five chapters that follow are four answers to a single question – what structure should be imposed on the weight matrix? – and one answer to a different question altogether.

Chapter 10 ties the weights and makes them local, and proves in Section Equivariance, and what it does and does not give that the result commutes with translation, so that a feature learned in one place is available everywhere. Chapter 11 ties them across time instead, and proves that the gradients then decay exponentially with lag, which is why long-memory tasks fail even when the architecture could express the answer, and why the gated cells of Section Long short-term memory were invented [hochreiter1997]. Chapter 12 removes the labels and asks the network to reproduce its own input through a narrow layer, and proves in Section The theorem that in the linear case this recovers exactly the principal subspace of Chapter 1 – which gives the exact floor against which any nonlinear autoencoder must be measured. Chapter 13 computes the weights from the data rather than storing them, and is equivariant under all permutations, a far larger group, which is why position has to be handed back to it explicitly in Section Masking and position.

Chapter 9 is the different question. It keeps the network and changes what is asked of it: instead of fitting data, the network is differentiated with respect to its own input and inserted into a differential equation, so that the residual becomes the cost function. Exponential decay, logistic growth, the Poisson equation, the diffusion and wave equations and a Black-Scholes problem are solved this way, and Section When is this worth doing? is honest about when this is worth doing and when a finite-difference scheme is simply better. Physics-informed neural networks [raissi2019, karniadakis2021] are developed in Section Physics-informed neural networks.

Part IV: generative methods#

The last four chapters change the question. Everything before them predicts a target given an input; these model the distribution of the data itself.

That change costs one thing, and it is worth naming precisely because the whole part is organised around it: a probability model needs a normalising constant, and that constant is a sum or an integral over a space too large to touch. It cannot be computed and it will not go away. What can be done is to find a formulation in which it is not needed, and the four chapters are four such formulations.

Chapter 14 refuses to compute it and samples instead. The gradient of the log-likelihood is a difference of two expectations, neither of which requires the value of the partition function, and Metropolis and Gibbs supply the necessary samples using ratios alone. The price is measured in Section How biased is CD-\texorpdfstring{\(k\)}{k}?: contrastive divergence with a single step gives a gradient that is biased, and by how much is stated there rather than glossed over. Chapter 15 bounds it, replacing the log-likelihood by the evidence lower bound of Section The evidence lower bound, and then measures the size of the gap it has introduced. Chapter 16 sidesteps it twice over, in one direction by learning a score, in which the normalisation has been differentiated away, and in the other by restricting to invertible maps whose Jacobian determinant is cheap [ho2020,dinh2017]. Chapter 17 abandons likelihood altogether and trains a classifier in its place [goodfellow2014gan]; the price is that the training problem becomes a game rather than a minimisation, and games need not converge.

Four evasions, four different prices. A reader who has understood the obstruction will be able to place the next generative model to appear by asking which of the four routes it takes, and what it therefore has to give up.

Part V and the concluding chapter#

Part V is reserved for reinforcement learning, which learns from interaction rather than from a fixed data set, and which needs a treatment of its own – Markov decision processes, the Bellman equations, temporal-difference learning and policy gradients. Sutton and Barto [sutton2018] remains the standard reference, and Chapter 18 says what connects it to the material developed here.

Chapter 18 is not a summary. It sets out the shape of the whole argument and the dependencies between its parts as a graph in Section The shape of the argument; it follows the handful of ideas that recur – conditioning, the derived loss, bias and variance, the chain rule, structure as a prior, the singular value decomposition, the intractable normalisation – and argues in Section The threads that these are the part of the material with the longest shelf life; it turns them round in Section A protocol for a new problem into a working order for attacking a problem that has not been seen before; it says plainly, in Section What this book does not contain, what the book does not contain; and it closes with a survey of where these methods are being taken in the physical sciences and with a list, in Section What to be sceptical about, of things to be sceptical about, drawn wherever possible from results in this book that did not come out as expected.

Readers in a hurry may reasonably read that chapter first.

How to read this book#

Every chapter is built the same way. The theory is developed with the intermediate algebra shown; each result that can be checked numerically is checked, by a program that is printed in the text and shipped with the book; each chapter closes with a section called Summary and the programs, which lists what was established and which file establishes it, and with a set of exercises graded from warm-up to project scale.

The dependencies are real but not oppressive. Chapters 1 and 2 are used by everything. Chapters 3-4 are used by everything after them. Chapter 8 is required for all of Parts III and IV. Within Part III, Chapters 9, 10, 11, 12 and 13 are independent of one another and may be read in any order or omitted. Within Part IV, Chapter 12 is a prerequisite for Chapter 15, and Chapter 15 for Chapter 16; Chapters 14 and 17 can be read on their own.

Three routes through the material have been used in practice.

A first course in data analysis and machine learning (one semester): Chapters 1-5, then Chapter 7 and Chapter 8, ending with Chapter 10. The support vector machines of Chapter 6 are then optional.
A course on deep learning for readers who already have the statistics: a rapid pass through Chapters 3 and 4, then Chapters 8-13 in full, followed by as much of Part IV as time allows.
Generative modelling and the physical sciences: Chapters 1, 2, 4 and 8 as prerequisites, then Chapter 12 and the whole of Part IV, with Chapter 9 added for readers interested in differential equations.

Notation is fixed once, in Section Notation and conventions, and adhered to throughout: vectors and matrices are boldface, vectors lower case and matrices upper case, and the design matrix is always \(\bm{X}\) with rows indexing samples and columns indexing features. Where a chapter needs a symbol that clashes with an earlier one, the clash is pointed out rather than silently tolerated.

Choice of programming language#

Python plays a central role in the development of machine learning techniques and tools for data analysis. The wealth of libraries written in Python, the ease with which they visualise results, the impressive galleries of existing examples, and the popularity of the Jupyter notebook framework – with the possibility of running compiled programs written in C++ or Fortran from within a notebook – made the choice of programming language for this book an easy one.

However, since the focus here is not only on using existing libraries such as Scikit-Learn, TensorFlow and PyTorch, but also on developing our own algorithms and codes, we present most algorithms first as plain Python built on NumPy, and only afterwards in the library form. Chapters 10, 11, 13, 14, 15 and 17 each carry a section in which the same model is written a second and a third time, in PyTorch and in TensorFlow, and the outputs are compared numerically against the implementation from scratch. The point of that exercise is not to teach an API. It is to make visible which parts of a library call correspond to which lines of the derivation, so that when a library result looks wrong the reader knows where to look.

Where computational speed genuinely matters we also give C++ or Fortran versions, written in an object-oriented style. A high-level language such as Python is excellent at structuring the flow of a calculation and at visualising it, while repetitive and CPU-intensive work is best left to a compiled language; a typical program in practice combines the two. Being multilingual is an advantage that applies to computing environments as much as it does to our globalised society.

The programs, the figures and the notebooks#

A rule has been followed throughout, and it is worth stating so that the reader can hold the book to it: every number quoted in the text is produced by a program printed in the text, and every figure is generated by running that program. No figure has been drawn by hand and no result has been quoted from a paper without being re-derived here.

This has three practical consequences. The programs are collected in the BookPrograms directory of the book’s source tree, one subdirectory per chapter, extracted verbatim from the chapters themselves, so that the LaTeX source and the runnable code cannot drift apart. The figures are generated by the scripts in BookFigures, with the same seeds as the programs, so that a figure and the number in the sentence beside it agree. And everything runs on a laptop in minutes – deliberately, because a result that the reader cannot reproduce is a result the reader has to take on trust.

The same material is available as an executable Jupyter-book, in which each chapter appears as a notebook and the parameters of every experiment can be varied. The book and the notebooks are maintained as parallel tracks covering the same subject matter; the printed book is self-contained, and a reader who never opens a notebook loses nothing but the ability to experiment conveniently.

On the results that did not come out well

A number of the experiments in this book fail, and the failures have been kept. A mode-collapsing generative model that covers fifteen of twenty-five modes, a recurrent network that cannot learn a dependency at lag one hundred, a constructed network that gradient descent cannot match, a bound that turns out to be several nats loose – these are reported as measured. They are the most useful numbers in the book, because they mark where the theory stops being a good description of practice, and a reader who has seen only successful runs has no calibration at all.

Data handling, machine learning and ethical aspects#

In most of the cases we study we either generate the data ourselves or use data sets distributed with Scikit-Learn or TensorFlow. Many of the examples we deal with are, from a privacy and data-protection point of view, rather innocuous and even boring results of numerical calculations. This does not absolve us from developing a sound ethical attitude to the data we use, to how we analyse them and to how we handle them.

The most immediate and simplest ethical aspects concern our approach to the scientific process itself. With version control software such as Git and online repositories such as GitHub and GitLab, we can easily make the codes and data sets we have used freely and openly accessible to a wider community. This helps us almost automatically in making our science reproducible. The large open-source communities behind Scikit-Learn, TensorFlow, PyTorch and Keras are excellent examples: the codes can be tested and improved upon continuously, which helps the scientific community at large. It is much easier today to gain traction and acceptance for making one’s science reproducible than it was a decade ago. From a societal standpoint this matters, since many developers are employees of public institutions such as universities and research laboratories. Our fellow taxpayers deserve to get something back for their money.

This more mechanical aspect of the ethics of science – the reproducibility of results – ought to be obvious, and it is part of the dialectics of science. The fact that many scientists are still unwilling to share their codes or their data is detrimental to the scientific discourse.

Before we proceed, a disclaimer is in order. Even though we may dream of computers developing some kind of higher learning capability, in the end it is we, yes you reading these lines, who construct and instruct, via various algorithms, the machine learning approaches. Self-driving cars, for example, rely on sophisticated programs which take into account the situations a car can encounter; extensive training data from GPS information and maps are fed into the software, and sensors and cameras feed further information to the programs. A great many ethical issues arise from this.

For self-driving cars, in which many of the standard machine learning algorithms discussed here enter the codes, at a certain stage we have to make choices – we, the people who wrote the program for a specific brand of car. All carmakers have as their utmost priority the security of the driver and the accompanying passengers. Suppose there are two obstacles ahead and a collision with one of them cannot be avoided. One obstacle is a heavy truck, the other a group of children crossing the road. An algorithm that ranks the survival probability of the occupants above everything else would opt for the children, since the likelihood of surviving a collision with the truck is much lower.

This leads to serious ethical questions. Why should we opt for such a rule? Who decides, and who is entitled to make such choices? Keep in mind that many of the algorithms you will encounter in this book, or hear about later, are based on simple programming instructions, and that you are very likely to be one of the people who end up writing such a code. Developing a sound ethical attitude to what we do, an approach well beyond the simple mechanistic one of making our science available and reproducible, is much needed. The self-driving car is just one of very many cases where we have to make choices. When you analyse data on economic inequality, who guarantees that you are not weighting some data in a particular way, perhaps because you would dearly like a specific conclusion which supports your political views? Or what about the recurring claims that the credit-scoring algorithms of large technology companies carry a gender bias?

There is a technical counterpart to all of this, and it is not separate from the ethical one. A model reproduces the distribution it was trained on. If that distribution encodes a historical inequity, so will the model, and the bias-variance decomposition of Section The bias-variance tradeoff says nothing whatever about it, because it measures the error with respect to the data generating process and not with respect to the world we should like to have. Nothing in this book will detect such a problem for you. What the book can do, and tries to do, is to insist that you know what your model assumes, what data it saw, and what it would do on data it did not see; a model whose assumptions can be named is a model whose failures can be explained. Section What to be sceptical about returns to this.

We do not have the answers here, nor shall we venture into a deeper discussion of these aspects, but we do want the reader to think about these topics in a more overarching way. A statistical data analysis, with its dry numbers and its graphs meant to guide the eye, does not necessarily reflect the truth, whatever that is. As a scientist, and after a university education, you are supposedly a better citizen, with an improved critical view and understanding of the scientific method, and perhaps some deeper understanding of the ethics of science at large. Use these insights. Be a critical citizen. You owe it to our society.