Session 19¶

New topic¶

Topic 3: Neural networks and PyTorch¶

• introduction to PyTorch
• automatic differentiation
• basic neural network models and training
• optimization hyperparameters
• distributed training with data parallelism
• distributed training on multiple compute nodes

Today¶

Introduction to PyTorch¶

1. what is PyTorch?
2. PyTorch tensors

Introduction to PyTorch¶

What is PyTorch?¶

an open-source Python library for

• tensor computation and
• automatic differentiation

developed by Meta AI (2016) and now the dominant framework in ML research

💡 like NumPy — but tensors can live on a GPU, and every operation is differentiable.

Why not just NumPy?¶

NumPy is excellent for array math, but it has two critical gaps for machine learning:

• no GPU support — NumPy always runs on the CPU
• no automatic differentiation — computing gradients requires manual calculus

PyTorch closes both gaps while keeping a NumPy-compatible API. 😎

Dynamic computation graphs¶

Most early frameworks (e.g. TensorFlow 1.x) required you to declare a computation graph before running it — a rigid, compile-then-execute model.

PyTorch builds the graph as your code runs:

• debugging with standard Python tools (print, pdb)
• control flow (if, for) that depends on data values
• faster conversion from idea to working experiment

The PyTorch ecosystem¶

four modules cover almost everything:

What’s next¶

we will build up from the foundation:

1. tensors — the core data structure 👈 today
2. autograd — how PyTorch computes derivatives automatically

… everything else in the ecosystem (nn, optim) sits on top of these two primitives; let’s get started

PyTorch tensors¶

What is a tensor? – core concept¶

A tensor is a multi-dimensional array.

💡 like NumPy arrays but with support GPUs and AD

• GPU: Graphics Processing Unit
• AD: Automatic Differentiation

Creating tensors¶

import torch

x = torch.empty(3, 4)

• creates a 2D tensor (3×4)
• values are uninitialized (random memory)

common alternatives with initial values:

a = torch.zeros(2, 3)  # filled with zeros
b = torch.ones(2, 3)   # filled with ones
c = torch.rand(2, 3)   # filled with random values from std. normal

→ more: Creation Ops

Random seeds for reproducibility:

a = torch.rand(2, 3)

→ a will have different values in each run of this code

torch.manual_seed(1729)
b = torch.rand(2, 3)

→ b will have the same values in each run of this code

• ensures repeatable experiments
• critical for: debugging, comparison of results between different runs

Tensor shapes – understanding dimensions¶

x = torch.empty(2, 2, 3)
print(f"dimension of x: {x.ndim}, shape of x: {x.shape}")

dimension of x: 3, shape of x: torch.Size([2, 2, 3])

meaning:

• 3D tensor
• structure matters for operations

⚠️ many operations require matching shapes

Creating tensors from other types¶

my_tensor_from_list = torch.tensor([[1, 2], [3, 4]])

my_np_array = np.zeros((100, 1, 28, 28))
my_tensor_from_np = torch.from_numpy(my_np_array)

• convert lists/tuples → tensors (copy of the data)
• convert NumPy arrays → tensors (same memory)

→ useful when importing data into PyTorch

Important: copy vs reference¶

b = a          # same memory
c = a.clone()  # new copy

• assignment → shared data
• .clone() → independent copy

Data types (dtype) control precision¶

a = torch.empty((2, 3))  # use the default: dtype=torch.float32
b = torch.rand((2, 3), dtype=torch.float64)
x = torch.ones((2, 3), dtype=torch.int16)

convert to a new dtype:

x.to(torch.int32)

important for:

• memory efficiency (lower precision is better)
• numerical stability (higher precision is better)

Tensor operations¶

Element-wise arithmetic¶

operations apply element by element / pointwise

a = torch.ones(2, 2)
b = torch.ones(2, 2) * 2

a + b
a * b
a * 10.0
b ** 2

💡 works like in NumPy

Tensor compatibility – shape matters!¶

a = torch.rand(2, 3)
b = torch.rand(3, 2)

a * b  # ❌ error

RuntimeError: The size of tensor a (3) must match the size of 
tensor b (2) at non-singleton dimension 1

• shapes must match for operations
• otherwise → runtime error

Broadcasting is shape matching¶

PyTorch automatically expands dimensions

a = torch.rand(2, 4)
b = torch.ones(1, 4)

a * b  # ✔️ ok

Rules:

• either: dimensions match exactly
• or: one dimension is 1

Operations beyond basic arithmetic¶

• math functions: torch.abs, torch.sin, torch.clamp, …
  → more: Pointwise Ops
• reductions: torch.mean, torch.max, torch.std, …
  → more: Reduction Ops
• linear algebra: torch.matmul, torch.svd, torch.linalg.norm, …
  → more: BLAS and LAPACK Ops, Linear Algebra Ops

In-place operations¶

modifies tensor directly (memory optimization)

a.add_(b)  # overwrites `a` with `a + b`

the suffix “_” means in-place

⚠️ use carefully: saves memory but can affect gradients

→ more: In-place operations with autograd

Slicing tensors¶

Accessing data in tensors¶

we often need to access specific entries of a tensor for

• data processing,
• feature selection,
• mini-batching, …

PyTorch tensors support NumPy-like indexing:

• extract subsets of data
• works across any number of dimensions

Basic Indexing – selecting elements¶

x = torch.tensor([[1, 2, 3],
                  [4, 5, 6]])

access elements:

x[0, 0]  # 1
x[1, 2]  # 6

→ syntax: tensor[row, column]

Slicing ranges – subsets of data¶

x = torch.tensor([[1, 2, 3],
                  [4, 5, 6]])

examples:

x[:, :]    # entire tensor
x[0, :]    # first row
x[:, 1]    # second column
x[:, 1:]   # columns 1 onward

→ syntax: start:end (end is exclusive)

Slicing 3D tensors¶

x = torch.rand(2, 3, 4)

example:

x[0, :, :]   # first "matrix"
x[:, 1, :]   # second row across all batches

💡 think in layers: batch × rows × columns

Skipping elements¶

x = torch.arange(10)

examples:

x[::2]   # every 2nd element
x[1::2]  # odd indices

→ syntax: start:end:step

Negative indexing¶

Counting from the end¶

x = torch.tensor([10, 20, 30, 40])

examples:

x[-1]  # 40
x[-2]  # 30

useful for:

• last elements
• reverse traversal

Important: slicing vs copy¶

x = torch.tensor([1, 2, 3, 4])
y = x[1:3]

• y is a view, not a copy

y[0] = 100  # changes the original `x`, too
print(x)    # prints "[1, 100, 3, 4]"

⚠️ slices share memory

Creating independent copies¶

y = x[1:3].clone()

• now y is independent
• changes won’t affect x

use when: you need safe modifications of a slice

Key takeaways¶

• PyTorch tensors = NumPy++ 🚀
  multi-dim arrays + GPUs + automatic gradients

• dynamic graphs = code like normal Python 🐍
  just write & run (debugging included 😌)

• shapes will bite you ⚠️
  most bugs caused by dimension mismatch → learn broadcasting or suffer

• Memory traps are real 👀

• slicing is a view (shared data!)

• .clone() is a real copy
• “_” ops are fast but can break gradients

Looking ahead¶

how neural networks are able to learn:

• automatic differentiation
• PyTorch autograd

Prepare for the next session 🌱¶

sessions/20-todo