Session 22¶

Topic 3: Neural networks and PyTorch¶

• introduction to PyTorch
• automatic differentiation
• basic neural network models and training
• optimization hyperparameters 👈
• distributed training with data parallelism
• distributed training on multiple compute nodes

Today¶

Optimization hyperparameters¶

1. learning rate and batch size
2. number of epochs and early stopping

Optimization hyperparameters¶

From learning → understanding¶

so far:

• tensors … store data
• autograd … automatic differentiation
• models … parameterized functions
• training … how the model learns

→ now we ask: why does training sometimes work…
and sometimes fail?

The hidden knobs¶

training is controlled by hyperparameters 🎛️

• they are not learned from data
• but chosen by us

important examples:

• learning rate
• batch size
• number of epochs
• early stopping criteria

Why should we care?¶

same model, same data, but…
other hyperparameters give completely different results

such as:

• loss converges fast ⚡
• loss converges slowly 🐢
• loss diverges 💥
• overfits training data ❌

Recall: the training loop¶

→ hyperparameters control how this loop behaves

The training loop in code¶

train_loader = DataLoader(..., batch_size=256)  # ← set batch size
optimizer = SGD(model.parameters(), lr=0.001)   # ← set learning rate

for epoch in range(num_epochs):                 # ← set num. epochs
    for i, data in enumerate(training_loader):
        # perform one training step

    if ...:
        break                                   # ← early stopping

Demo: training on MNIST dataset¶

code/pytorch/mnist

the “hello world” of neural networks

• dataset: handwritten digits (0–9)
• grayscale images (28 × 28 pixels)
• 10 classes → classification problem

Goal: train a simple convolutional network that recognizes digits from images

Learning rate¶

optimizer = SGD(model.parameters(), lr=0.001)

learning rate means step size of optimization

• too small → very slow learning 😴
• too large → unstable / diverging 💥

→ most important hyperparameter

Batch size¶

train_loader = DataLoader(..., batch_size=256)

how many samples per gradient descent update

• small batch → noisy gradients 🎲
• large batch → stable but expensive 💰

→ trade-off: speed (increase batch size) vs generalization (decrease batch size)

Number of epochs¶

for epoch in range(num_epochs):
    ...

balance how long we train:

• too few epochs → underfitting
• too many epochs → overfitting

💡 comes from the neural nets ability to approximate any mapping (features → targets)

Early stopping¶

a validation data set (not used for optimization):

• stop when validation accuracy is “good enough”
• and stays stable for a few epochs

(“good enough” is another hyperparameter)

→ prevents wasting time + reduces overfitting

What we will explore 🔬¶

you will run experiments by changing:

• learning rate
• batch size
• number of epochs
• early stopping settings

and observe:

• training speed
• stability
• final accuracy

Key idea¶

training is a dynamical process and hyperparameters shape:

• the path through parameter space
• the speed of convergence
• the final solution

Your hands-on tasks¶

code/pytorch/fashion_mnist

don’t just run the code, but probe it, try:

• batch_size = 8 … 512
• fewer / more epochs with early stopping on / off
• lr = 0.001 vs 0.0001

ask:

• what changed?
• and why?

Key takeaways¶

• models learn from data
• but training behavior depends on hyperparameters

→ understanding them gives you control over training

Looking ahead¶

from learning → scaling

• distributed training with PyTorch DDP
• converting single-GPU code to multi-GPU

Prepare for the next session 🌱¶

sessions/23-todo