Project 2¶

Distributed Data Parallel Training on CIFAR-10¶

In this project you will take a provided single-GPU training script for a WideResNet classifier on the CIFAR-10 dataset and parallelize it using PyTorch's Distributed Data Parallel (DDP) framework. You will deploy and benchmark your implementation on two NVIDIA A30 GPUs on Virginia Tech's ARC cluster, Falcon.

This project brings together everything covered in the lecture series: tensor operations, autograd, model construction, the training loop, optimization hyperparameters, and — centrally — distributed data parallel training.

Learning Objectives¶

By the end of this project you should be able to:

• Understand the multi-process execution model underlying DDP.
• Correctly initialize and tear down a distributed process group.
• Partition data across workers using DistributedSampler.
• Wrap a model with DistributedDataParallel and pin each process to its assigned GPU.
• Synchronize metrics (loss, accuracy, throughput) across ranks.
• Reason about the performance implications of your design choices.

Provided Materials¶

You are given:

• cifar10.py — a fully working single-GPU training script that trains a WideResNet on CIFAR-10. This is your starting point.
• A reference DDP implementation for the Fashion-MNIST dataset (posted on the course page). This is a reference only — you are not submitting it and should not copy it blindly. Study it to understand the DDP idioms, then apply those idioms to the CIFAR-10 script.
• A Slurm job script template for Falcon (job_template.sh at docs/assignments/projects).

Deliverables¶

Submit the following to the course submission portal by the deadline.

Do not submit dataset files or Python virtual environments.

Cluster Setup (Falcon)¶

Use the provided Slurm template job_template.sh at docs/assignments/projects. We provide a detailed description how to use PyTorch on Falcon. Remember to benchmark your code on two A30 GPUs.

Tips¶

• Start early. Cluster jobs can queue for hours during peak periods.
• Test locally first. Run with just 1 GPU on Google Colab (or your own machine if you have a GPU) to catch bugs before requesting a multi-GPU allocation on Falcon.
• Read error messages carefully. Most DDP initialization failures come from mismatched WORLD_SIZE / rank values or an unreachable MASTER_ADDR.
• Use dist.barrier() deliberately. Place barriers only where synchronization is semantically required; excessive barriers hurt performance.
• Profile before optimizing. For the challenge, measure where time is actually spent (data loading, forward pass, backward pass, communication) before guessing what to change.

General Instructions¶

All submitted work must be stored in your team's GitLab repository. Below it will be explained how to do that. For the repository it is important that:

1. Create a new repository on GitLab at code.vt.edu:

   • one project = one GitLab repository → each project gets its own GitLab repository
   • all team members of your project must be members of the repository
   • your repository must be private
   • your repository must be named exactly as specified: do not change the name or the upper-case/lower-case letters

     cmda3634-pr2
     

   • create commits frequently, at least one commit per task is recommended
   • push your work frequently (this also is your backup if your laptop breaks)
   • your repository must be up-to-date before the deadline
   • you must double-check your repository in the web browser to make sure all your work is uploaded to the code.vt.edu server

2. Add the instructor and TA to your repository as a member with Developer access. In the web browser with your new repository open:

   • in the left sidebar, hover the mouse cursor over "Manage"
   • in the menu that pops up, click "Members"; a new page loads
   • on the new page, click button "Invite members"
     • enter the jrudi
     • select maximum role: "Developer"
     • click the button "Invite"
   • repeat to invite a new member:
     • enter the jraind
     • select maximum role: "Developer"
     • click the button "Invite"

⚠️ You must run your code yourself! Not following this is an Honor Code violation.

⚠️ You must create complete log files of all AI conversations! Not following this is an Honor Code violation.

Requirements¶

Your submission must satisfy all of the following.

R1 — Correct DDP Integration (50 points)¶

Modify cifar10.py so that training can be distributed across an arbitrary number of GPUs and nodes. Concretely, your script must:

1. Accept command-line arguments for --num-nodes, --node-id, and --num-gpus.
2. Compute the correct WORLD_SIZE and each process's global_rank.
3. Initialize the process group with dist.init_process_group (backend "nccl").
4. Download the dataset on only one process per node; use dist.barrier() to synchronize before other processes open the dataset.
5. Create a DistributedSampler for both the training and validation sets, and pass the appropriate sampler to each DataLoader.
6. Call train_sampler.set_epoch(epoch) at the start of every epoch to ensure different shuffle orderings across epochs.
7. Assign each process to the correct GPU ("cuda:<local_rank>").
8. Wrap the model with nn.parallel.DistributedDataParallel.
9. After each training epoch, reduce images_per_sec to rank 0 with dist.reduce; use dist.all_reduce with dist.ReduceOp.AVG to average validation loss and accuracy across all ranks.
10. Print per-epoch metrics (cumulative time, epoch time, images/sec, validation loss, validation accuracy) from rank 0 only.
11. Launch worker processes with torch.multiprocessing.spawn inside the if __name__ == '__main__': guard.

R2 — Target Accuracy (20 points)¶

Your training run on two A30 GPUs must reach a validation accuracy ≥ 0.85 for at least two consecutive epochs before stopping. You may use the provided early- stopping logic or implement your own.

R3 — Report (20 points)¶

Submit a short written report (PDF, ≤ 4 pages) that includes:

• A brief description of each DDP change you made and why it is necessary.
• The printed epoch log from your successful two-GPU run (copy-paste or screenshot).
• Your total cumulative training time at the point the stopping criterion was first met.
• A discussion of any optimizations you applied (see Challenge below).

Challenge: Fastest Training Time (up to 10 bonus points)¶

This part is optional but encouraged.

Once your implementation is correct, optimize it for wall-clock time to reach validation accuracy ≥ 0.85 for two consecutive epochs on exactly two A30 GPUs.

You are free to modify anything in the training algorithm: the learning rate schedule, batch size, optimizer, data augmentation, mixed-precision training, SyncBatchNorm usage, gradient compression, or anything else. The model architecture (WideResNet) and the dataset (CIFAR-10) must remain unchanged.

The top-performing submissions will be re-run by the instructors in a controlled benchmark. Bonus points (up to 10) are awarded based on your cumulative training time relative to the other submissions. The fastest valid run receives the full 10 points; all other valid runs receive a fractional score computed from the time difference using an undisclosed formula.

In your report, dedicate a section to the challenge. For each optimization you applied, state:

1. what you changed,
2. why you expected it to help, and
3. the measured impact (before/after cumulative time or images/sec).

If you don't document an optimization, it will be turned off in the instructor's benchmark.

Demonstrate (in your documentation and project presentation)¶

1. Process Topology and Rank Assignment¶

Using a diagram, show your node/GPU layout and explain how global_rank is computed from node_id and local_rank. Point to the lines of code that pin each process to its GPU and initialize the process group.

2. Data Partitioning¶

Explain what DistributedSampler does and why each process must see a distinct shard. Explain why train_sampler.set_epoch(epoch) is required each epoch.

3. Gradient Synchronization¶

Explain conceptually what DistributedDataParallel does during the backward pass: when the all-reduce fires, what is averaged, and why all replicas stay identical after each optimizer step.

4. Metric Aggregation¶

Show your aggregation code and explain why validation accuracy/loss uses ReduceOp.AVG while images_per_sec uses a sum. What does the summed throughput represent physically?

5. Correctness and Scaling¶

Present the epoch log from your two-GPU run on Falcon. In a table, compare 1 GPU vs. 2 GPUs: cumulative time to stopping criterion, images/sec, and final validation accuracy. Relate the observed speedup to Amdahl's law and identify any bottleneck.

6. Optimizations (Challenge participants)¶

For each optimization: state what you changed, why you expected it to help, and provide before/after numbers (time, images/sec, or epochs to convergence). Report negative results honestly.