GPU Lab GPU map
Open a component to explore its connections. Follow chapter links for explanations and experiments.
Computer / system
Open the GPU chip, then an SM. Keep opening components to see blocks, warps and threads.
Host (CPU side)
In CUDA, “host” means the CPU side of the system; “device” means the GPU accelerator.
CPU (Ch. 2)Runs your host program
The CPU runs the Python or C++ host program and arranges GPU work. Its hardware caches are part of the processor; system RAM is separate.
Book example: Calling a PyTorch operation can arrange asynchronous GPU work. Copying its result to the CPU waits for the work needed to produce that result.
The CPU reads and writes system memory.
System memory / RAM (Ch. 2)Holds CPU tensor data · separate from the CPU
System RAM holds CPU tensor storage in these examples. It is part of the host side of the computer, alongside the CPU rather than inside it.
Book example: The chapter starts with a CPU tensor and later copies GPU results back so the CPU can use them.
GPU device / accelerator
A discrete NVIDIA GPU device includes a compute chip and dedicated memory. It may be a card or a module.
GPU chip (Ch. 2, 6)Contains SMs, caches and memory controllersInside: Ch. 1–2, 5–10 · extra practice
The compute chip contains many streaming multiprocessors (SMs). Each SM has its own L1 cache; the SMs share an on-chip L2 cache.
SM 0 (Ch. 6)Streaming multiprocessor · 2 resident blocks shownInside: Ch. 1, 5–10 · extra practice
An SM is a part of the GPU that executes instructions for resident threads. It can host several blocks when resources allow.
Ready work and scheduling (Ch. 6)Selects ready instructions from warps
An SM selects ready warp instructions to issue. Dependencies can make work wait.
Book example: In the teaching scheduler, another ready warp can issue while one waits for a load. Its toy cycles are not hardware timings.
Registers (Ch. 6, 9)Working values private to each thread
Per-thread working storage on an SM. Register use can constrain how much work can reside there.
Book example: Increasing a tile size can increase both reuse and resource demand; larger is not automatically faster.
L1 cache (Ch. 5)On-chip cache local to this SM
A hardware cache local to an SM. It can retain data used by that SM’s threads.
Book example: Unlike shared memory, which the kernel uses explicitly, cached data is managed by hardware. On many NVIDIA architectures, L1 and shared memory share on-chip capacity.
Tensor cores (Ch. 10)Specialised units for supported matrix operations
Specialised execution units for supported matrix operations, shapes and numerical formats.
Book example: Changing dtype does not guarantee tensor-core use. The chapter’s sequential accumulation model is not tensor-core instruction execution.
Work assigned to this SM
Block A (Ch. 1, 8–9)Resident work · 64 threads → 2 warpsInside: Ch. 5–9 · extra practice
A block is a group of threads that can cooperate. It stays on this SM for its lifetime.
Warp 0 (Ch. 5–7)Threads 0–31 · 32 lane positionsInside: Ch. 5, 7
A lane is a thread’s position within its warp. T labels give the thread index within block A. Select a lane to inspect its thread.
Lane 0 T0 (Ch. 5, 7)
Lane 0 → thread 0 within block A. Its registers hold its own working values.
Lane 1 T1 (Ch. 5, 7)
Lane 1 → thread 1 within block A. Its registers hold its own working values.
Lane 2 T2 (Ch. 5, 7)
Lane 2 → thread 2 within block A. Its registers hold its own working values.
Lane 3 T3 (Ch. 5, 7)
Lane 3 → thread 3 within block A. Its registers hold its own working values.
Lane 4 T4 (Ch. 5, 7)
Lane 4 → thread 4 within block A. Its registers hold its own working values.
Lane 5 T5 (Ch. 5, 7)
Lane 5 → thread 5 within block A. Its registers hold its own working values.
Lane 6 T6 (Ch. 5, 7)
Lane 6 → thread 6 within block A. Its registers hold its own working values.
Lane 7 T7 (Ch. 5, 7)
Lane 7 → thread 7 within block A. Its registers hold its own working values.
Lane 8 T8 (Ch. 5, 7)
Lane 8 → thread 8 within block A. Its registers hold its own working values.
Lane 9 T9 (Ch. 5, 7)
Lane 9 → thread 9 within block A. Its registers hold its own working values.
Lane 10 T10 (Ch. 5, 7)
Lane 10 → thread 10 within block A. Its registers hold its own working values.
Lane 11 T11 (Ch. 5, 7)
Lane 11 → thread 11 within block A. Its registers hold its own working values.
Lane 12 T12 (Ch. 5, 7)
Lane 12 → thread 12 within block A. Its registers hold its own working values.
Lane 13 T13 (Ch. 5, 7)
Lane 13 → thread 13 within block A. Its registers hold its own working values.
Lane 14 T14 (Ch. 5, 7)
Lane 14 → thread 14 within block A. Its registers hold its own working values.
Lane 15 T15 (Ch. 5, 7)
Lane 15 → thread 15 within block A. Its registers hold its own working values.
Lane 16 T16 (Ch. 5, 7)
Lane 16 → thread 16 within block A. Its registers hold its own working values.
Lane 17 T17 (Ch. 5, 7)
Lane 17 → thread 17 within block A. Its registers hold its own working values.
Lane 18 T18 (Ch. 5, 7)
Lane 18 → thread 18 within block A. Its registers hold its own working values.
Lane 19 T19 (Ch. 5, 7)
Lane 19 → thread 19 within block A. Its registers hold its own working values.
Lane 20 T20 (Ch. 5, 7)
Lane 20 → thread 20 within block A. Its registers hold its own working values.
Lane 21 T21 (Ch. 5, 7)
Lane 21 → thread 21 within block A. Its registers hold its own working values.
Lane 22 T22 (Ch. 5, 7)
Lane 22 → thread 22 within block A. Its registers hold its own working values.
Lane 23 T23 (Ch. 5, 7)
Lane 23 → thread 23 within block A. Its registers hold its own working values.
Lane 24 T24 (Ch. 5, 7)
Lane 24 → thread 24 within block A. Its registers hold its own working values.
Lane 25 T25 (Ch. 5, 7)
Lane 25 → thread 25 within block A. Its registers hold its own working values.
Lane 26 T26 (Ch. 5, 7)
Lane 26 → thread 26 within block A. Its registers hold its own working values.
Lane 27 T27 (Ch. 5, 7)
Lane 27 → thread 27 within block A. Its registers hold its own working values.
Lane 28 T28 (Ch. 5, 7)
Lane 28 → thread 28 within block A. Its registers hold its own working values.
Lane 29 T29 (Ch. 5, 7)
Lane 29 → thread 29 within block A. Its registers hold its own working values.
Lane 30 T30 (Ch. 5, 7)
Lane 30 → thread 30 within block A. Its registers hold its own working values.
Lane 31 T31 (Ch. 5, 7)
Lane 31 → thread 31 within block A. Its registers hold its own working values.
Warp 1 (Ch. 5–7)Threads 32–63 · 32 lane positionsInside: Ch. 5, 7
A lane is a thread’s position within its warp. T labels give the thread index within block A. Select a lane to inspect its thread.
Lane 0 T32 (Ch. 5, 7)
Lane 0 → thread 32 within block A. Its registers hold its own working values.
Lane 1 T33 (Ch. 5, 7)
Lane 1 → thread 33 within block A. Its registers hold its own working values.
Lane 2 T34 (Ch. 5, 7)
Lane 2 → thread 34 within block A. Its registers hold its own working values.
Lane 3 T35 (Ch. 5, 7)
Lane 3 → thread 35 within block A. Its registers hold its own working values.
Lane 4 T36 (Ch. 5, 7)
Lane 4 → thread 36 within block A. Its registers hold its own working values.
Lane 5 T37 (Ch. 5, 7)
Lane 5 → thread 37 within block A. Its registers hold its own working values.
Lane 6 T38 (Ch. 5, 7)
Lane 6 → thread 38 within block A. Its registers hold its own working values.
Lane 7 T39 (Ch. 5, 7)
Lane 7 → thread 39 within block A. Its registers hold its own working values.
Lane 8 T40 (Ch. 5, 7)
Lane 8 → thread 40 within block A. Its registers hold its own working values.
Lane 9 T41 (Ch. 5, 7)
Lane 9 → thread 41 within block A. Its registers hold its own working values.
Lane 10 T42 (Ch. 5, 7)
Lane 10 → thread 42 within block A. Its registers hold its own working values.
Lane 11 T43 (Ch. 5, 7)
Lane 11 → thread 43 within block A. Its registers hold its own working values.
Lane 12 T44 (Ch. 5, 7)
Lane 12 → thread 44 within block A. Its registers hold its own working values.
Lane 13 T45 (Ch. 5, 7)
Lane 13 → thread 45 within block A. Its registers hold its own working values.
Lane 14 T46 (Ch. 5, 7)
Lane 14 → thread 46 within block A. Its registers hold its own working values.
Lane 15 T47 (Ch. 5, 7)
Lane 15 → thread 47 within block A. Its registers hold its own working values.
Lane 16 T48 (Ch. 5, 7)
Lane 16 → thread 48 within block A. Its registers hold its own working values.
Lane 17 T49 (Ch. 5, 7)
Lane 17 → thread 49 within block A. Its registers hold its own working values.
Lane 18 T50 (Ch. 5, 7)
Lane 18 → thread 50 within block A. Its registers hold its own working values.
Lane 19 T51 (Ch. 5, 7)
Lane 19 → thread 51 within block A. Its registers hold its own working values.
Lane 20 T52 (Ch. 5, 7)
Lane 20 → thread 52 within block A. Its registers hold its own working values.
Lane 21 T53 (Ch. 5, 7)
Lane 21 → thread 53 within block A. Its registers hold its own working values.
Lane 22 T54 (Ch. 5, 7)
Lane 22 → thread 54 within block A. Its registers hold its own working values.
Lane 23 T55 (Ch. 5, 7)
Lane 23 → thread 55 within block A. Its registers hold its own working values.
Lane 24 T56 (Ch. 5, 7)
Lane 24 → thread 56 within block A. Its registers hold its own working values.
Lane 25 T57 (Ch. 5, 7)
Lane 25 → thread 57 within block A. Its registers hold its own working values.
Lane 26 T58 (Ch. 5, 7)
Lane 26 → thread 58 within block A. Its registers hold its own working values.
Lane 27 T59 (Ch. 5, 7)
Lane 27 → thread 59 within block A. Its registers hold its own working values.
Lane 28 T60 (Ch. 5, 7)
Lane 28 → thread 60 within block A. Its registers hold its own working values.
Lane 29 T61 (Ch. 5, 7)
Lane 29 → thread 61 within block A. Its registers hold its own working values.
Lane 30 T62 (Ch. 5, 7)
Lane 30 → thread 62 within block A. Its registers hold its own working values.
Lane 31 T63 (Ch. 5, 7)
Lane 31 → thread 63 within block A. Its registers hold its own working values.
Block C (Ch. 1, 8–9)Resident work · 64 threads → 2 warpsInside: Ch. 5–9 · extra practice
A block is a group of threads that can cooperate. It stays on this SM for its lifetime.
Warp 0 (Ch. 5–7)Threads 0–31 · 32 lane positionsInside: Ch. 5, 7
A lane is a thread’s position within its warp. T labels give the thread index within block C. Select a lane to inspect its thread.
Lane 0 T0 (Ch. 5, 7)
Lane 0 → thread 0 within block C. Its registers hold its own working values.
Lane 1 T1 (Ch. 5, 7)
Lane 1 → thread 1 within block C. Its registers hold its own working values.
Lane 2 T2 (Ch. 5, 7)
Lane 2 → thread 2 within block C. Its registers hold its own working values.
Lane 3 T3 (Ch. 5, 7)
Lane 3 → thread 3 within block C. Its registers hold its own working values.
Lane 4 T4 (Ch. 5, 7)
Lane 4 → thread 4 within block C. Its registers hold its own working values.
Lane 5 T5 (Ch. 5, 7)
Lane 5 → thread 5 within block C. Its registers hold its own working values.
Lane 6 T6 (Ch. 5, 7)
Lane 6 → thread 6 within block C. Its registers hold its own working values.
Lane 7 T7 (Ch. 5, 7)
Lane 7 → thread 7 within block C. Its registers hold its own working values.
Lane 8 T8 (Ch. 5, 7)
Lane 8 → thread 8 within block C. Its registers hold its own working values.
Lane 9 T9 (Ch. 5, 7)
Lane 9 → thread 9 within block C. Its registers hold its own working values.
Lane 10 T10 (Ch. 5, 7)
Lane 10 → thread 10 within block C. Its registers hold its own working values.
Lane 11 T11 (Ch. 5, 7)
Lane 11 → thread 11 within block C. Its registers hold its own working values.
Lane 12 T12 (Ch. 5, 7)
Lane 12 → thread 12 within block C. Its registers hold its own working values.
Lane 13 T13 (Ch. 5, 7)
Lane 13 → thread 13 within block C. Its registers hold its own working values.
Lane 14 T14 (Ch. 5, 7)
Lane 14 → thread 14 within block C. Its registers hold its own working values.
Lane 15 T15 (Ch. 5, 7)
Lane 15 → thread 15 within block C. Its registers hold its own working values.
Lane 16 T16 (Ch. 5, 7)
Lane 16 → thread 16 within block C. Its registers hold its own working values.
Lane 17 T17 (Ch. 5, 7)
Lane 17 → thread 17 within block C. Its registers hold its own working values.
Lane 18 T18 (Ch. 5, 7)
Lane 18 → thread 18 within block C. Its registers hold its own working values.
Lane 19 T19 (Ch. 5, 7)
Lane 19 → thread 19 within block C. Its registers hold its own working values.
Lane 20 T20 (Ch. 5, 7)
Lane 20 → thread 20 within block C. Its registers hold its own working values.
Lane 21 T21 (Ch. 5, 7)
Lane 21 → thread 21 within block C. Its registers hold its own working values.
Lane 22 T22 (Ch. 5, 7)
Lane 22 → thread 22 within block C. Its registers hold its own working values.
Lane 23 T23 (Ch. 5, 7)
Lane 23 → thread 23 within block C. Its registers hold its own working values.
Lane 24 T24 (Ch. 5, 7)
Lane 24 → thread 24 within block C. Its registers hold its own working values.
Lane 25 T25 (Ch. 5, 7)
Lane 25 → thread 25 within block C. Its registers hold its own working values.
Lane 26 T26 (Ch. 5, 7)
Lane 26 → thread 26 within block C. Its registers hold its own working values.
Lane 27 T27 (Ch. 5, 7)
Lane 27 → thread 27 within block C. Its registers hold its own working values.
Lane 28 T28 (Ch. 5, 7)
Lane 28 → thread 28 within block C. Its registers hold its own working values.
Lane 29 T29 (Ch. 5, 7)
Lane 29 → thread 29 within block C. Its registers hold its own working values.
Lane 30 T30 (Ch. 5, 7)
Lane 30 → thread 30 within block C. Its registers hold its own working values.
Lane 31 T31 (Ch. 5, 7)
Lane 31 → thread 31 within block C. Its registers hold its own working values.
Warp 1 (Ch. 5–7)Threads 32–63 · 32 lane positionsInside: Ch. 5, 7
A lane is a thread’s position within its warp. T labels give the thread index within block C. Select a lane to inspect its thread.
Lane 0 T32 (Ch. 5, 7)
Lane 0 → thread 32 within block C. Its registers hold its own working values.
Lane 1 T33 (Ch. 5, 7)
Lane 1 → thread 33 within block C. Its registers hold its own working values.
Lane 2 T34 (Ch. 5, 7)
Lane 2 → thread 34 within block C. Its registers hold its own working values.
Lane 3 T35 (Ch. 5, 7)
Lane 3 → thread 35 within block C. Its registers hold its own working values.
Lane 4 T36 (Ch. 5, 7)
Lane 4 → thread 36 within block C. Its registers hold its own working values.
Lane 5 T37 (Ch. 5, 7)
Lane 5 → thread 37 within block C. Its registers hold its own working values.
Lane 6 T38 (Ch. 5, 7)
Lane 6 → thread 38 within block C. Its registers hold its own working values.
Lane 7 T39 (Ch. 5, 7)
Lane 7 → thread 39 within block C. Its registers hold its own working values.
Lane 8 T40 (Ch. 5, 7)
Lane 8 → thread 40 within block C. Its registers hold its own working values.
Lane 9 T41 (Ch. 5, 7)
Lane 9 → thread 41 within block C. Its registers hold its own working values.
Lane 10 T42 (Ch. 5, 7)
Lane 10 → thread 42 within block C. Its registers hold its own working values.
Lane 11 T43 (Ch. 5, 7)
Lane 11 → thread 43 within block C. Its registers hold its own working values.
Lane 12 T44 (Ch. 5, 7)
Lane 12 → thread 44 within block C. Its registers hold its own working values.
Lane 13 T45 (Ch. 5, 7)
Lane 13 → thread 45 within block C. Its registers hold its own working values.
Lane 14 T46 (Ch. 5, 7)
Lane 14 → thread 46 within block C. Its registers hold its own working values.
Lane 15 T47 (Ch. 5, 7)
Lane 15 → thread 47 within block C. Its registers hold its own working values.
Lane 16 T48 (Ch. 5, 7)
Lane 16 → thread 48 within block C. Its registers hold its own working values.
Lane 17 T49 (Ch. 5, 7)
Lane 17 → thread 49 within block C. Its registers hold its own working values.
Lane 18 T50 (Ch. 5, 7)
Lane 18 → thread 50 within block C. Its registers hold its own working values.
Lane 19 T51 (Ch. 5, 7)
Lane 19 → thread 51 within block C. Its registers hold its own working values.
Lane 20 T52 (Ch. 5, 7)
Lane 20 → thread 52 within block C. Its registers hold its own working values.
Lane 21 T53 (Ch. 5, 7)
Lane 21 → thread 53 within block C. Its registers hold its own working values.
Lane 22 T54 (Ch. 5, 7)
Lane 22 → thread 54 within block C. Its registers hold its own working values.
Lane 23 T55 (Ch. 5, 7)
Lane 23 → thread 55 within block C. Its registers hold its own working values.
Lane 24 T56 (Ch. 5, 7)
Lane 24 → thread 56 within block C. Its registers hold its own working values.
Lane 25 T57 (Ch. 5, 7)
Lane 25 → thread 57 within block C. Its registers hold its own working values.
Lane 26 T58 (Ch. 5, 7)
Lane 26 → thread 58 within block C. Its registers hold its own working values.
Lane 27 T59 (Ch. 5, 7)
Lane 27 → thread 59 within block C. Its registers hold its own working values.
Lane 28 T60 (Ch. 5, 7)
Lane 28 → thread 60 within block C. Its registers hold its own working values.
Lane 29 T61 (Ch. 5, 7)
Lane 29 → thread 61 within block C. Its registers hold its own working values.
Lane 30 T62 (Ch. 5, 7)
Lane 30 → thread 62 within block C. Its registers hold its own working values.
Lane 31 T63 (Ch. 5, 7)
Lane 31 → thread 63 within block C. Its registers hold its own working values.
A block’s threads do not necessarily execute all at once. Ready work from other resident blocks may proceed while threads wait or lanes are unused. Unused lanes cannot be filled by another block’s threads.
SM 1 (Ch. 6)Streaming multiprocessor · 1 resident block shownInside: Ch. 1, 5–10 · extra practice
An SM is a part of the GPU that executes instructions for resident threads. It can host several blocks when resources allow.
Ready work and scheduling (Ch. 6)Selects ready instructions from warps
An SM selects ready warp instructions to issue. Dependencies can make work wait.
Book example: In the teaching scheduler, another ready warp can issue while one waits for a load. Its toy cycles are not hardware timings.
Registers (Ch. 6, 9)Working values private to each thread
Per-thread working storage on an SM. Register use can constrain how much work can reside there.
Book example: Increasing a tile size can increase both reuse and resource demand; larger is not automatically faster.
L1 cache (Ch. 5)On-chip cache local to this SM
A hardware cache local to an SM. It can retain data used by that SM’s threads.
Book example: Unlike shared memory, which the kernel uses explicitly, cached data is managed by hardware. On many NVIDIA architectures, L1 and shared memory share on-chip capacity.
Tensor cores (Ch. 10)Specialised units for supported matrix operations
Specialised execution units for supported matrix operations, shapes and numerical formats.
Book example: Changing dtype does not guarantee tensor-core use. The chapter’s sequential accumulation model is not tensor-core instruction execution.
Work assigned to this SM
Block B (Ch. 1, 8–9)Resident work · 64 threads → 2 warpsInside: Ch. 5–9 · extra practice
A block is a group of threads that can cooperate. It stays on this SM for its lifetime.
Warp 0 (Ch. 5–7)Threads 0–31 · 32 lane positionsInside: Ch. 5, 7
A lane is a thread’s position within its warp. T labels give the thread index within block B. Select a lane to inspect its thread.
Lane 0 T0 (Ch. 5, 7)
Lane 0 → thread 0 within block B. Its registers hold its own working values.
Lane 1 T1 (Ch. 5, 7)
Lane 1 → thread 1 within block B. Its registers hold its own working values.
Lane 2 T2 (Ch. 5, 7)
Lane 2 → thread 2 within block B. Its registers hold its own working values.
Lane 3 T3 (Ch. 5, 7)
Lane 3 → thread 3 within block B. Its registers hold its own working values.
Lane 4 T4 (Ch. 5, 7)
Lane 4 → thread 4 within block B. Its registers hold its own working values.
Lane 5 T5 (Ch. 5, 7)
Lane 5 → thread 5 within block B. Its registers hold its own working values.
Lane 6 T6 (Ch. 5, 7)
Lane 6 → thread 6 within block B. Its registers hold its own working values.
Lane 7 T7 (Ch. 5, 7)
Lane 7 → thread 7 within block B. Its registers hold its own working values.
Lane 8 T8 (Ch. 5, 7)
Lane 8 → thread 8 within block B. Its registers hold its own working values.
Lane 9 T9 (Ch. 5, 7)
Lane 9 → thread 9 within block B. Its registers hold its own working values.
Lane 10 T10 (Ch. 5, 7)
Lane 10 → thread 10 within block B. Its registers hold its own working values.
Lane 11 T11 (Ch. 5, 7)
Lane 11 → thread 11 within block B. Its registers hold its own working values.
Lane 12 T12 (Ch. 5, 7)
Lane 12 → thread 12 within block B. Its registers hold its own working values.
Lane 13 T13 (Ch. 5, 7)
Lane 13 → thread 13 within block B. Its registers hold its own working values.
Lane 14 T14 (Ch. 5, 7)
Lane 14 → thread 14 within block B. Its registers hold its own working values.
Lane 15 T15 (Ch. 5, 7)
Lane 15 → thread 15 within block B. Its registers hold its own working values.
Lane 16 T16 (Ch. 5, 7)
Lane 16 → thread 16 within block B. Its registers hold its own working values.
Lane 17 T17 (Ch. 5, 7)
Lane 17 → thread 17 within block B. Its registers hold its own working values.
Lane 18 T18 (Ch. 5, 7)
Lane 18 → thread 18 within block B. Its registers hold its own working values.
Lane 19 T19 (Ch. 5, 7)
Lane 19 → thread 19 within block B. Its registers hold its own working values.
Lane 20 T20 (Ch. 5, 7)
Lane 20 → thread 20 within block B. Its registers hold its own working values.
Lane 21 T21 (Ch. 5, 7)
Lane 21 → thread 21 within block B. Its registers hold its own working values.
Lane 22 T22 (Ch. 5, 7)
Lane 22 → thread 22 within block B. Its registers hold its own working values.
Lane 23 T23 (Ch. 5, 7)
Lane 23 → thread 23 within block B. Its registers hold its own working values.
Lane 24 T24 (Ch. 5, 7)
Lane 24 → thread 24 within block B. Its registers hold its own working values.
Lane 25 T25 (Ch. 5, 7)
Lane 25 → thread 25 within block B. Its registers hold its own working values.
Lane 26 T26 (Ch. 5, 7)
Lane 26 → thread 26 within block B. Its registers hold its own working values.
Lane 27 T27 (Ch. 5, 7)
Lane 27 → thread 27 within block B. Its registers hold its own working values.
Lane 28 T28 (Ch. 5, 7)
Lane 28 → thread 28 within block B. Its registers hold its own working values.
Lane 29 T29 (Ch. 5, 7)
Lane 29 → thread 29 within block B. Its registers hold its own working values.
Lane 30 T30 (Ch. 5, 7)
Lane 30 → thread 30 within block B. Its registers hold its own working values.
Lane 31 T31 (Ch. 5, 7)
Lane 31 → thread 31 within block B. Its registers hold its own working values.
Warp 1 (Ch. 5–7)Threads 32–63 · 32 lane positionsInside: Ch. 5, 7
A lane is a thread’s position within its warp. T labels give the thread index within block B. Select a lane to inspect its thread.
Lane 0 T32 (Ch. 5, 7)
Lane 0 → thread 32 within block B. Its registers hold its own working values.
Lane 1 T33 (Ch. 5, 7)
Lane 1 → thread 33 within block B. Its registers hold its own working values.
Lane 2 T34 (Ch. 5, 7)
Lane 2 → thread 34 within block B. Its registers hold its own working values.
Lane 3 T35 (Ch. 5, 7)
Lane 3 → thread 35 within block B. Its registers hold its own working values.
Lane 4 T36 (Ch. 5, 7)
Lane 4 → thread 36 within block B. Its registers hold its own working values.
Lane 5 T37 (Ch. 5, 7)
Lane 5 → thread 37 within block B. Its registers hold its own working values.
Lane 6 T38 (Ch. 5, 7)
Lane 6 → thread 38 within block B. Its registers hold its own working values.
Lane 7 T39 (Ch. 5, 7)
Lane 7 → thread 39 within block B. Its registers hold its own working values.
Lane 8 T40 (Ch. 5, 7)
Lane 8 → thread 40 within block B. Its registers hold its own working values.
Lane 9 T41 (Ch. 5, 7)
Lane 9 → thread 41 within block B. Its registers hold its own working values.
Lane 10 T42 (Ch. 5, 7)
Lane 10 → thread 42 within block B. Its registers hold its own working values.
Lane 11 T43 (Ch. 5, 7)
Lane 11 → thread 43 within block B. Its registers hold its own working values.
Lane 12 T44 (Ch. 5, 7)
Lane 12 → thread 44 within block B. Its registers hold its own working values.
Lane 13 T45 (Ch. 5, 7)
Lane 13 → thread 45 within block B. Its registers hold its own working values.
Lane 14 T46 (Ch. 5, 7)
Lane 14 → thread 46 within block B. Its registers hold its own working values.
Lane 15 T47 (Ch. 5, 7)
Lane 15 → thread 47 within block B. Its registers hold its own working values.
Lane 16 T48 (Ch. 5, 7)
Lane 16 → thread 48 within block B. Its registers hold its own working values.
Lane 17 T49 (Ch. 5, 7)
Lane 17 → thread 49 within block B. Its registers hold its own working values.
Lane 18 T50 (Ch. 5, 7)
Lane 18 → thread 50 within block B. Its registers hold its own working values.
Lane 19 T51 (Ch. 5, 7)
Lane 19 → thread 51 within block B. Its registers hold its own working values.
Lane 20 T52 (Ch. 5, 7)
Lane 20 → thread 52 within block B. Its registers hold its own working values.
Lane 21 T53 (Ch. 5, 7)
Lane 21 → thread 53 within block B. Its registers hold its own working values.
Lane 22 T54 (Ch. 5, 7)
Lane 22 → thread 54 within block B. Its registers hold its own working values.
Lane 23 T55 (Ch. 5, 7)
Lane 23 → thread 55 within block B. Its registers hold its own working values.
Lane 24 T56 (Ch. 5, 7)
Lane 24 → thread 56 within block B. Its registers hold its own working values.
Lane 25 T57 (Ch. 5, 7)
Lane 25 → thread 57 within block B. Its registers hold its own working values.
Lane 26 T58 (Ch. 5, 7)
Lane 26 → thread 58 within block B. Its registers hold its own working values.
Lane 27 T59 (Ch. 5, 7)
Lane 27 → thread 59 within block B. Its registers hold its own working values.
Lane 28 T60 (Ch. 5, 7)
Lane 28 → thread 60 within block B. Its registers hold its own working values.
Lane 29 T61 (Ch. 5, 7)
Lane 29 → thread 61 within block B. Its registers hold its own working values.
Lane 30 T62 (Ch. 5, 7)
Lane 30 → thread 62 within block B. Its registers hold its own working values.
Lane 31 T63 (Ch. 5, 7)
Lane 31 → thread 63 within block B. Its registers hold its own working values.
A block’s threads do not necessarily execute all at once. Ready work from other resident blocks may proceed while threads wait or lanes are unused. Unused lanes cannot be filled by another block’s threads.
SMs read and write through the on-chip cache hierarchy.
L2 cache (Ch. 5)Inside the chip · shared across SMsInside: Ch. 5 · extra practice
An on-chip hardware cache shared across the SMs. It can retain data from device memory and reduce repeated accesses to that memory.
Book example: The memory-request chapter counts addressed sectors, not cache misses or DRAM reads. It does not model cache contents or hit rates.
Sectors and coalescing (Ch. 5)Definition, book example and chapter links
Nearby addresses requested by a warp can share aligned memory regions. The book groups FP32 reads into 32-byte sectors.
Book example: Five neighbouring values can cross a sector boundary. Sector coverage is not a count of physical DRAM reads or a speed measurement.
Memory controllers (Ch. 2, 5)Inside the chip · connect to device memory
On-chip hardware that manages reads and writes to the GPU device’s dedicated memory.
Book example: An SM’s memory request may be served by a cache. Accesses to device memory go through memory controllers; the browser does not model their timing.
Two SMs are shown. Hardware manages caches; a kernel uses each block’s shared-memory workspace explicitly. On many NVIDIA architectures, L1 and shared memory share on-chip capacity. Boxes show relationships, not the chip’s physical layout.
Memory controllers connect the chip to its dedicated memory.
Device memory / VRAM (Ch. 2, 5)Inside the GPU device · separate from the compute chipInside: Ch. 1–4, 10
Dedicated memory, often called VRAM, holds CUDA tensor storage. It is part of the GPU device but separate from the compute chip: typically GDDR on the board or HBM stacks alongside the GPU die in the package.
Book example: Different blocks can read the same input tensor. Shared memory is a separate, block-scoped workspace.
Tensor (Ch. 1–3)Definition, book example and chapter linksInside: Ch. 2–4, 10
A collection of values described by properties including shape, data type, device and strides.
Book example: The same five values can be stored on the CPU or GPU. Moving or viewing them is different from changing their values.
Shape and slicing (Ch. 3)Definition, book example and chapter links
Shape gives dimension sizes; slicing selects values along those dimensions.
Book example: A 5 × 5 tensor has five rows and five columns. Selecting its first column gives values 1, 6, 11, 16 and 21.
Strides and views (Ch. 4)Definition, book example and chapter links
Strides locate elements in storage. A view accesses existing storage instead of copying its values.
Book example: The column view steps five elements at a time; a transpose changes how the same storage is interpreted.
Data type and precision (Ch. 2, 10)Definition, book example and chapter links
The representation of each value affects storage size, range and rounding.
Book example: FP32 occupies four bytes. Converting an input to FP16 or BF16 can lose information before accumulation begins.
Copies and transfers (Ch. 2, 4)Definition, book example and chapter links
A copy creates values in another storage allocation. A transfer moves data between devices.
Book example: Copying the inputs to the GPU preserves the CPU tensor. Packing the column creates contiguous storage but also costs reads and writes.
Illustrative placement: three blocks of 64 threads on two SMs. Each block stays on one SM; several blocks may share it. This is a relationship map, not a schedule or a promise that every thread executes at once. Integrated GPUs can share system memory; this picture shows a discrete GPU device.