24 - Quiz¶

• Your name:

• Your VT email:

What is the main idea behind Distributed Data Parallel training?

• A. Split the model across GPUs so each GPU computes part of the forward pass
• B. Use a single GPU but parallelize matrix operations internally
• C. Replicate the model on each GPU and split the data across processes
• D. Store the dataset on multiple disks to speed up loading

How does DDP keep model parameters consistent across GPUs?

• A. By copying the model from GPU 0 after each epoch
• B. By averaging gradients across all processes during backpropagation
• C. By sharing a single global optimizer across processes
• D. By saving and reloading checkpoints after every batch

In DDP, which statement is true about model parameters?

• A. Each GPU has different parameters at all times
• B. Only one GPU stores parameters; others fetch them
• C. All processes maintain synchronized copies of the model
• D. Parameters are stored only on the CPU

If training becomes faster with more GPUs but validation accuracy drops, what is a likely reason?

• A. The dataset is corrupted
• B. The model cannot run on multiple GPUs
• C. Hyperparameters (e.g., learning rate) were not adjusted
• D. Gradient synchronization failed intermittently