GPU Lab CHAPTER 4
How are tensor values arranged in memory?
Follow values into storage, compare strides, then make a packed copy.
The same values, seen in memory
PyTorch
import torch
x = torch.arange(1, 26, dtype=torch.float32, device="cuda").reshape(5, 5)
values = x[:, 0]
y = values * 2We use the same 5 × 5 tensor containing 1 through 25. x[:, 0] selects the first column: [1, 6, 11, 16, 21]. Selecting a column chooses values without moving them. To understand how PyTorch finds them, we need to follow the grid into the line of values stored in memory.
Definitions
- Row-major order
- An order that stores the first row, then the second row, and so on.
- Contiguous layout
- A layout with values next to one another in the tensor’s expected order. Our original grid is contiguous in row-major order.
Each example below starts with this x and has its own controls and result. PyTorch handles address calculations; your choice of view or copy affects which storage it reads.
Run the examples below in your browser with no installation or GPU required. The browser simulates the calculation; it does not execute PyTorch or GPU code. To run the PyTorch snippet itself, use a Python environment with PyTorch and an NVIDIA GPU.
Idea 1 of 3
Follow a row into memory
Definitions
- View
- A tensor that accesses existing storage rather than copying its values.
- Stride
- The number of element positions to move when an index increases by one.
- Offset
- A distance from the start of storage, counted in elements or bytes as labelled.
x[0, :] is a view. Its values occupy neighbouring positions in x’s storage, so its stride is 1. A value and its offset are different: the value 1 is at element offset 0.
Try it
Predict the element offsets for row 0. Run, then select row 1. Follow the highlighted values into the memory strip.
PyTorch · operation for these controls
values = x[0, :]
y = values * 2Predict the outputs, then press Run.
Browser simulation · this example keeps its own controls and results.
x: five rows, five columns
Highlighted cells supply the five inputs.
View into x’s storage
- Selected value 0
1Element ?
Byte offset ?× 2 →
? - Selected value 1
2Element ?
Byte offset ?× 2 →
? - Selected value 2
3Element ?
Byte offset ?× 2 →
? - Selected value 3
4Element ?
Byte offset ?× 2 →
? - Selected value 4
5Element ?
Byte offset ?× 2 →
?
x in memory order
Read left to right, then continue on the next line. Small labels are element offsets from the start of this storage.
The next value in a row is one element away. Each FP32 element takes four bytes, so that step is four bytes.
Idea 2 of 3
Follow a column into the same memory
x[:, 0] is also a view. The next value in a column is five element positions away, so its stride is 5. The original numbers stay where they are.
Try it
Predict the element offsets for column 0. Run, then choose column 1. Compare the gaps with the row example.
PyTorch · operation for these controls
values = x[:, 0]
y = values * 2Predict the outputs, then press Run.
Browser simulation · this example keeps its own controls and results.
x: five rows, five columns
Highlighted cells supply the five inputs.
View into x’s storage
- Selected value 0
1Element ?
Byte offset ?× 2 →
? - Selected value 1
6Element ?
Byte offset ?× 2 →
? - Selected value 2
11Element ?
Byte offset ?× 2 →
? - Selected value 3
16Element ?
Byte offset ?× 2 →
? - Selected value 4
21Element ?
Byte offset ?× 2 →
?
x in memory order
Read left to right, then continue on the next line. Small labels are element offsets from the start of this storage.
Five FP32 elements span 20 bytes. PyTorch uses strides to locate the values; selecting a column does not move those values next to one another.
Idea 3 of 3
Make a packed copy
Calling .contiguous() on this column creates a separate copy with its five values next to one another. PyTorch can multiply the original column view directly; packing is an optional layout choice.
Try it
Predict whether packing changes the numbers. Run and compare with the column view above. Then pack another column.
PyTorch · operation for these controls
column = x[:, 0]
packed = column.contiguous()
y = packed * 2Predict the outputs, then press Run.
Browser simulation · this example keeps its own controls and results.
x: five rows, five columns
Highlighted cells supply the five inputs.
New packed storage
- Selected value 0
1Element ?
Byte offset ?× 2 →
? - Selected value 1
6Element ?
Byte offset ?× 2 →
? - Selected value 2
11Element ?
Byte offset ?× 2 →
? - Selected value 3
16Element ?
Byte offset ?× 2 →
? - Selected value 4
21Element ?
Byte offset ?× 2 →
?
Packed copy in memory order
Read left to right, then continue on the next line. Small labels are element offsets from the start of this storage.
The inputs and results match the column view, but the offsets differ. Packing reads and writes data. It helps only when later benefits outweigh that copy cost; it is not automatically faster. Calling .contiguous() on an already-contiguous tensor can return it without a copy.
Explain what you found
Try a new case
Would calling .contiguous() on x[0, :] need to copy its values? Explain using the memory strip.
Compare your explanation
This row already has contiguous values with stride 1, so .contiguous() can return it without making a copy.
Optional practice: transpose the grid
transposed = x.T
y = transposed[0, :] * 2.T swaps the row and column interpretation. Predict the first row and whether the values move.
Compare your explanation
The first row is [1, 6, 11, 16, 21]. The transpose is a view: it shares storage. Its strides are (1, 5), whereas x’s are (5, 1). The values do not move.
Run the example on your desktop
Download this chapter’s opening PyTorch example. See the complete example index and setup appendix for CPU PyTorch, the supported GPGPU-Sim programs and optional NVIDIA GPU execution.
The browser still requires no installation. Downloads are generated from the displayed source; CPU checks and simulator checks do not establish GPU performance.
References
Use these official references to check the concepts and code on this page.
- PyTorch: tensor views (new tab)
Distinguish changing how values are accessed from copying them.
- PyTorch: Tensor.stride (new tab)
Check the unit of a stride.
- PyTorch: Tensor.contiguous (new tab)
Check when requesting a contiguous layout makes a copy.
These browser exercises do not run an installed PyTorch or CUDA toolkit. For optional local execution, choose documentation matching your installed tools. Links open online in a new tab; the questions and answer comparisons below work offline.
Use the documentation
Use the references to explain each answer, then apply it to this page’s example. No installation is needed.