Session 20¶

Topic 3: Neural networks and PyTorch¶

introduction to PyTorch
automatic differentiation 👈
basic neural network models and training
optimization hyperparameters
distributed training with data parallelism
distributed training on multiple compute nodes

Today¶

Automatic differentiation¶

what is automatic differentiation?
PyTorch autograd

Automatic differentiation¶

What is automatic differentiation?¶

Automatic Differentiation (AD) are techniques to evaluate derivatives of functions defined by computer programs

exactly
efficiently
without symbolic manipulation

AD is not:

symbolic differentiation (no expression blowup)
finite differences (no truncation error, no step-size tuning)

AD is:

the chain rule, applied systematically to every elementary operation in a program

Let’s start with an example¶

consider the function:

\[ \begin{aligned} y &= (\sin(x))^2 \\ \text{derivative} &= 2 \sin(x) \cos(x) \end{aligned} \]

let’s write it more general:

\[ \begin{aligned} F(x) &= f_2 \circ f_1 (x) = (\cdot)^2 \circ \sin(x) \\ F^\prime(x) &= f_2^\prime(f_1(x)) \times f_1^\prime(x) \end{aligned} \]

How the chain rule becomes an algorithm¶

compute the example function:

\[ x ~\rightarrow~ h_1 = \sin(x) ~\rightarrow~ h_2 = h_1^2 \]

compute the derivative using forward mode:

\[ \bar{h}_1 = \cos(x) ~\rightarrow~ \bar{h}_2 \times \bar{h}_1 = 2 \sin(x) \times \cos(x) \]

function output: \(~y = h_2\)
derivative output: \(~\bar{h}_2 \times \bar{h}_1\)

Another take with the chain rule¶

compute the example function:

\[ x ~\rightarrow~ h_1 = \sin(x) ~\rightarrow~ h_2 = h_1^2 \]

compute the derivative using reverse mode:

\[ \bar{h}_2 \times \bar{h}_1 = 2 \sin(x) \times \cos(x) ~\leftarrow~ \bar{h}_2 = 2 \sin(x) \]

function output: \(~y = h_2\)
derivative output: \(~\bar{h}_2 \times \bar{h}_1\)

General composition of 2 functions¶

function computation:

\[ x ~\rightarrow~ h_1 = f_1(x) ~\rightarrow~ h_2 = f_2(h_1) \]

forward mode differentiation: take some \(\dot{s}\)

\[ \dot{s} ~\rightarrow~ \bar{h}_1 \dot{s} = f_1^\prime(x) \dot{s} ~\rightarrow~ \bar{h}_2 \!\times\! \bar{h}_1 \dot{s} = f_2^\prime(h_1) \!\times\! f_1^\prime(x) \dot{s} \]

reverse mode differentiation: take some \(\bar{s}\)

\[ \bar{s}\, \bar{h}_2 \!\times\! \bar{h}_1 = \bar{s}\, f_2^\prime(h_1) \!\times\! f_1^\prime(x) ~\leftarrow~ \bar{s}\, \bar{h}_2 = \bar{s}\, f_2^\prime(h_1) ~\leftarrow~ \bar{s} \]

General automatic differentiation¶

view any program as a composition of elementary maps:

\[ F : \mathbb{R}^n \to \mathbb{R}^m, \quad F = f_p \circ f_{p-1} \circ \cdots \circ f_1 \]

define the intermediate values:

\[ V_0 = \text{input}, \quad V_k = f_k(V_{k-1}), \quad k = 1, \ldots, p \]

The chain rule¶

by the chain rule, the derivative of \(F\) is:

\[ F^\prime(V_0) = f^\prime_p(V_{p-1})\; f^\prime_{p-1}(V_{p-2})\; \cdots\; f^\prime_1(V_0) \]

💡 \(F^\prime(V_0)\) is called the Jacobian at \(V_0\)

→ \(F^\prime(V_0)\) is equal to a product of matrices

we (almost) never want to compute that product
instead, we compute matrix-vector products

Forward mode¶

for an arbitrary seed vector \(\dot{S} \in \mathbb{R}^n\), multiply from right to left:

\[ F^\prime(V_0) \, \dot{S} = \underbrace{f^\prime_p(V_{p-1})}_\text{end} \; f^\prime_{p-1}(V_{p-2}) \; \cdots \; \underbrace{f^\prime_1(V_0) \; \dot{S}}_\text{start} \]

Reverse mode¶

for an arbitrary seed vector \(\bar{S} \in \mathbb{R}^m\) (called co-vector), multiply from left to right:

\[ \bar{S}^T \, F^\prime(V_0) = \underbrace{\bar{S}^T f^\prime_p(V_{p-1})}_\text{start} \; f^\prime_{p-1}(V_{p-2}) \; \cdots \; \underbrace{f^\prime_1(V_0)}_\text{end} \]

Cost model: when to use which mode?¶

\(F : \mathbb{R}^n \to \mathbb{R}^m\) has the full Jacobian \(F^\prime \in \mathbb{R}^{m \times n}\)

Mode	Single pass computes	Full \(F^\prime\) requires	Preferred when
forward	one column of \(F^\prime\)	\(n\) passes	\(n \leq m\)
reverse	one row of \(F^\prime\)	\(m\) passes	\(m \leq n\)

The gradient is the poster child for reverse mode¶

if \(F : \mathbb{R}^n \to \mathbb{R}\) (i.e., \(m = 1\)), then

\[ F^\prime = \nabla F \in \mathbb{R}^n \]

→ recovered in a single reverse pass!

the forward mode would require \(n\) passes
for large \(n\) (e.g., neural network weights) the reverse mode is dramatically cheaper

PyTorch autograd¶

Deep learning needs gradients¶

machine learning means solving an optimization problem

How does loss change with respect to parameters?

we minimize a loss function to train an ML model
it requires gradients (derivatives)
gradients drive learning

What is autograd?¶

PyTorch’s engine for computing gradients with AD
tracks operations on tensors
builds a computation graph dynamically

→ computes derivatives using the chain rule

Tracking gradients (requires_grad)¶

enable gradient tracking:

import torch

x = torch.tensor(2.0, requires_grad=True)

PyTorch will:

record operations
compute gradients later

💡 needed for trainable parameters of an ML model

Building a computation graph¶

example:

x = torch.tensor(2.0, requires_grad=True)
y = x ** 2 + 3 * x

PyTorch creates a graph:

nodes = operations
edges = data flow

→ the graph is built on the fly (dynamically)

Backpropagation: computing gradients¶

y.backward()
print(x.grad)

computes \(\dfrac{dy}{dx}\) for the specific value of \(x\)

example:

\[ y = x^2 + 3x ~\rightarrow~ \frac{dy}{dx} = 2x + 3 ~\rightarrow~ \frac{dy}{dx}(2) = 7 \]

then evaluate at \(x = 2\)

Important behavior: gradients accumulate¶

y.backward()
y.backward()

gradients are accumulated, not overwritten

→ need to reset:

x.grad.zero_()

💡 remember resetting during training loops

Disabling gradient tracking¶

no gradient needed:

with torch.no_grad():
    y = x * 2

stops tracking computations
saves memory & computation

→ used for:

inference
evaluation

Detaching tensors¶

breaking the graph:

y = x.detach()

creates tensor without gradient history
shares data, but without gradient tracking

→ useful for:

logging
output intermediate results

Key takeaways¶

autograd = no more manual calculus 🙏
every operation tracked → gradients “just work”

key concepts:

requires_grad=True
computation graphs
.backward() for gradients
gradient accumulation
no_grad() and .detach()

Looking ahead¶

now we have the foundation for training neural networks

next time: let’s train them

Prepare for the next session 🌱¶

sessions/21-todo