GPU Lab CHAPTER 6
How can work continue during a wait?
Trace a load dependency, then see when another ready warp can start its next instruction.
Start with the operation
PyTorch
import torch
x = torch.tensor([1., 2., 3., 4., 5.], device="cuda")
y = x * 2
z = y * 2
print(z.cpu())The five inputs from the opening chapters are scaled twice: y contains the first products and z the second. You choose the tensors and operations; PyTorch selects implementations, allocates the results and arranges their launches. Copying z to the CPU lets Python print its values.
A calculation may have to wait for a loaded input. We will model that dependency with two repetitions of LD → MUL → ST: load, multiply, store. This is a deliberately chosen instruction stream, not a compiler trace of the PyTorch program. The model does not compute its tensor values or infer how PyTorch assigns them to warps.
Run the examples below in your browser with no installation or GPU required. The browser simulates the calculation; it does not execute PyTorch or GPU code. To run the PyTorch snippet itself, use a Python environment with PyTorch and an NVIDIA GPU.
Idea 1 of 2
Follow one warp through a load wait
Definitions
- Streaming multiprocessor (SM)
- A GPU unit on which a block’s threads run, grouped into warps. This model begins with warps assigned to one SM.
- Cycle
- One discrete step of this teaching machine, not a measured duration on a GPU.
- Load latency
- The number of model cycles from issuing a load until its value is ready for the dependent multiply.
Begin with one warp on an SM. A load starts a wait; its dependent multiply cannot start until the value is ready. In the diagram, ready means the next instruction can start; waiting means its input is still unavailable.
The chosen sequence is LD, MUL, ST, LD, MUL, ST. The teaching machine starts at most one instruction each cycle; an idle cycle starts none. A latency of 6 means a load started at cycle 0 becomes usable at cycle 6. Watch what can happen between those two events. The model ends when the last store starts; it does not model a later wait for the store to complete.
Try it
With one warp, predict where idle cycles appear. Run and step past the first load. Change only load latency to 1, run again, and explain what happened to the waits.
Predict the result, then press Run.
- Model cycles
- 16
- Instructions started
- 6
- Idle cycles
- 10
Cycle 0: warp 0 starts LD.
Warp states immediately before this cycle’s instruction starts. LD = load; MUL = multiply; ST = store.
Next: LD.
Inspect the complete model trace
| Cycle | Selected warp | Instruction | Waiting warps |
|---|---|---|---|
| 0 | 0 | LD | None |
| 1 | None | idle | 0 |
| 2 | None | idle | 0 |
| 3 | None | idle | 0 |
| 4 | None | idle | 0 |
| 5 | None | idle | 0 |
| 6 | 0 | MUL | None |
| 7 | 0 | ST | None |
| 8 | 0 | LD | None |
| 9 | None | idle | 0 |
| 10 | None | idle | 0 |
| 11 | None | idle | 0 |
| 12 | None | idle | 0 |
| 13 | None | idle | 0 |
| 14 | 0 | MUL | None |
| 15 | 0 | ST | None |
6 instructions start in 16 model cycles; 10 cycles are idle. Step through a load and its following multiply to locate the wait.
Idea 2 of 2
Give the scheduler another ready warp
One warp left gaps while waiting for data. Could the SM use those gaps for another warp? That requires other work already on the SM and ready to continue.
Definitions
- Resident warp
- A warp already assigned resources on an SM. Every warp in this experiment is resident from the start.
- Instruction issue
- Selecting a ready warp and starting its next instruction. An issue slot is one opportunity to do that; this teaching machine has one per cycle.
- Scheduler
- The mechanism that chooses which ready resident warp issues next. This teaching scheduler searches in round-robin order after the last warp selected.
- Issue-slot use
- Issued instructions divided by elapsed model cycles. This describes how many teaching issue slots were used.
All resident warps begin ready at cycle zero. After each issue, the scheduler searches from the next warp, skipping waiting and completed warps. When one waits for its load, another may have an instruction ready. The cards show each warp immediately before the selected cycle issues.
Keep latency fixed and change only the number of resident warps. Each extra warp adds another six instructions, so the total work changes too. Compare idle slots and issue-slot use; elapsed cycles for these different workloads do not give a speedup.
This teaching SM starts with its warps already resident and does not calculate resource limits or competition for memory bandwidth. Its single issue slot and fixed load latency describe this experiment, rather than a particular GPU.
Try it
Predict whether two ready warps can fill every wait at latency 6. Run, inspect an idle cycle if one remains, then change only the warp count to 4. Explain the difference using readiness and the extra work.
Predict the result, then press Run.
- Model cycles
- 20
- Issued instructions
- 12
- Idle cycles
- 8
- Issue-slot use
- 60%
Cycle 0: warp 0 issues LD.
Warp states immediately before this cycle’s instruction starts. LD = load; MUL = multiply; ST = store.
Next: LD.
Next: LD.
Inspect the complete model trace
| Cycle | Selected warp | Instruction | Waiting warps |
|---|---|---|---|
| 0 | 0 | LD | None |
| 1 | 1 | LD | 0 |
| 2 | None | idle | 0, 1 |
| 3 | None | idle | 0, 1 |
| 4 | None | idle | 0, 1 |
| 5 | None | idle | 0, 1 |
| 6 | 0 | MUL | 1 |
| 7 | 1 | MUL | None |
| 8 | 0 | ST | None |
| 9 | 1 | ST | None |
| 10 | 0 | LD | None |
| 11 | 1 | LD | 0 |
| 12 | None | idle | 0, 1 |
| 13 | None | idle | 0, 1 |
| 14 | None | idle | 0, 1 |
| 15 | None | idle | 0, 1 |
| 16 | 0 | MUL | 1 |
| 17 | 1 | MUL | None |
| 18 | 0 | ST | None |
| 19 | 1 | ST | None |
12 instructions issued in 20 model cycles; 8 slots were idle. Every resident warp adds six instructions. This is issue-slot use for the selected work, not occupancy or a measured GPU speedup.
Optional: place the warps in the bigger picture
The experiments began with warps already resident. This separate ownership map puts their blocks on two SMs. It uses three blocks of 64 threads, rather than the five tensor inputs in the opening. Inspect the block with two warps: warp membership follows thread indices, while available resources constrain which blocks can reside on an SM.
Inside a GPU: SMs → blocks → warps → threads
Separate placement example: 3 blocks, 64 threads per block.
SM 0
Threads 0–31 within block A
Threads 32–63 within block A
Threads 0–31 within block C
Threads 32–63 within block C
SM 1
Threads 0–31 within block B
Threads 32–63 within block B
Blocks can make progress independently. The SM can choose ready warps from any of the blocks it holds. Another block can run while some threads in the first are waiting or some lanes are unused, when resources allow. A block does not have to finish before another can start.
In chapter 1’s four-thread exercise, each block uses one partially filled warp: 4 of 32 lanes have threads. The other 28 positions do not prevent another block from running, and cannot be filled with threads from another block.
Try a new case
Apply it
A resident warp is waiting on a load, and another resident warp is ready. Must the next issue slot be idle? Explain what this model can tell you, and whether doubling the resident warp count proves a speedup.
Compare your answer
The ready warp can issue: the scheduler skips the waiting warp. Adding resident warps also adds six instructions per warp in this experiment. Compare issue-slot use and idle slots, but do not turn the elapsed cycles of different workloads into a speedup. Actual residency and performance need resource and hardware evidence beyond this model.
Run the example on your desktop
Download this chapter’s opening PyTorch example. See the complete example index and setup appendix for CPU PyTorch, the supported GPGPU-Sim programs and optional NVIDIA GPU execution.
The browser still requires no installation. Downloads are generated from the displayed source; CPU checks and simulator checks do not establish GPU performance.
References
Use these official references to check the concepts and code on this page.
- CUDA: hardware multithreading (new tab)
Distinguish resident work, ready work and hardware resource limits.
- CUDA: writing and launching kernels (new tab)
Review how launched threads are grouped into blocks.
These browser exercises do not run an installed PyTorch or CUDA toolkit. For optional local execution, choose documentation matching your installed tools. Links open online in a new tab; the questions and answer comparisons below work offline.
Use the documentation
Use the references to explain each answer, then apply it to this page’s example. No installation is needed.