GPU Lab CHAPTER 2
Where do the numbers live?
Follow the same five values from CPU memory to GPU memory and back.
The same calculation, with the journey made explicit
PyTorch
import torch
x_cpu = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0], dtype=torch.float32)
x_gpu = x_cpu.to("cuda")
y_gpu = x_gpu * 2
y_cpu = y_gpu.cpu()Chapter 1 created a tensor directly on the GPU. Here we start with a CPU tensor so we can follow the transfer. The suffixes _cpu and _gpu are names we chose to make each tensor’s location clear.
Definitions
- Device
- The processor location chosen for a tensor: CPU or GPU here.
- Storage
- The memory holding a tensor’s values.
- FP32
- A 32-bit floating-point number, taking four bytes of storage.
The setup uses dtype=torch.float32 explicitly for these four-byte values.
What do you control? You choose where tensors live and when to request a copy with .to("cuda") or .cpu(). PyTorch arranges the destination storage and transfer for these copies, and launches GPU computation when you operate on a CUDA tensor. It does not copy every result back to the CPU automatically. Keeping related operations on the GPU can avoid repeated transfers.
The three examples below each include the setup they need. Their diagrams show the same five values before and after one operation.
Run the examples below in your browser with no installation or GPU required. The browser simulates the calculation; it does not execute PyTorch or GPU code. To run the PyTorch snippet itself, use a Python environment with PyTorch and an NVIDIA GPU.
Idea 1 of 3
Copy the inputs to the GPU
Definitions
- RAM
- System memory holding the values of an ordinary CPU tensor.
- VRAM
- GPU memory holding the values of a CUDA tensor in these examples.
The CPU runs your Python program and asks the GPU to perform work. Memory holds the numbers; the processors do the calculation.
Try it
Predict: will copying the inputs to the GPU remove the CPU tensor? Run, then change the number of inputs and compare the copied bytes.
Setup for this example
import torch
x_cpu = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0], dtype=torch.float32)x_gpu = x_cpu.to("cuda")Predict the result, then press Run.
Browser simulation · this example keeps its own controls and results.
CPU · system RAM
x_cpu · 20 bytes
GPU · GPU memory (VRAM)
No example tensors here yet.
Before this operation. Run to see where the new tensor appears.
The CPU tensor remains. Because the device changes, .to("cuda") makes a GPU copy. The diagram counts tensor values, excluding metadata and allocator overhead.
Idea 2 of 3
Calculate where the inputs live
Multiplying a CUDA tensor produces a result on the GPU. Here y_gpu gets its own storage; the input x_gpu keeps its original values.
Try it
Predict: does this multiplication need to copy its result back to the CPU? Change the multiplier and run.
Setup for this example
import torch
x_cpu = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0], dtype=torch.float32)
x_gpu = x_cpu.to("cuda")y_gpu = x_gpu * 2Predict the result, then press Run.
Browser simulation · this example keeps its own controls and results.
CPU · system RAM
x_cpu · 20 bytes
GPU · GPU memory (VRAM)
x_gpu · 20 bytes
Before this operation. Run to see where the new tensor appears.
The GPU reads x_gpu and writes y_gpu in GPU memory. This operation copies zero bytes between CPU and GPU. It still reads and writes GPU memory; zero transfers does not mean zero work.
Idea 3 of 3
Bring the result back when you need it
Call .cpu() when you need a CPU copy of a GPU tensor. Here Python keeps a reference to that copy as y_cpu.
Try it
Predict where the results will appear. Run, then use a different multiplier and copy the new results.
Setup for this example
import torch
x_cpu = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0], dtype=torch.float32)
x_gpu = x_cpu.to("cuda")
y_gpu = x_gpu * 2y_cpu = y_gpu.cpu()Predict the result, then press Run.
Browser simulation · this example keeps its own controls and results.
CPU · system RAM
x_cpu · 20 bytes
GPU · GPU memory (VRAM)
x_gpu · 20 bytes
y_gpu · 20 bytes
Before this operation. Run to see where the new tensor appears.
The result is now available in system RAM as well as GPU memory. Copying does not delete y_gpu or change x_cpu. If your next operation also runs on the GPU, it can use y_gpu directly. This .cpu() call waits for the work needed to produce and copy the result, so the CPU can use the returned values.
Try a new case
Predict a new case
Suppose you run z_gpu = y_gpu + 1. Which tensor should you copy to the CPU to read the final answer? Would x_cpu change?
Compare your explanation
Copy z_gpu.cpu(). The original CPU input stays [1, 2, 3, 4, 5]. Keeping these operations on the GPU avoids an intermediate trip through CPU memory.
This model describes ordinary separate CPU and GPU allocations. It counts tensor bytes, not elapsed time, physical bus traffic, allocation reuse or overlapping work. Calling .to() with an unchanged device and dtype normally returns the same tensor; copying is not automatic on every call.
Run the example on your desktop
Download this chapter’s opening PyTorch example. See the complete example index and setup appendix for CPU PyTorch, the supported GPGPU-Sim programs and optional NVIDIA GPU execution.
The browser still requires no installation. Downloads are generated from the displayed source; CPU checks and simulator checks do not establish GPU performance.
References
Use these official references to check the concepts and code on this page.
- PyTorch: Tensor.to (new tab)
Check when a device conversion returns the original tensor or makes a copy.
- PyTorch: Tensor.cpu (new tab)
Check how a CPU copy is requested.
- PyTorch: CUDA semantics (new tab)
Read the opening rules for the device on which results are created.
These browser exercises do not run an installed PyTorch or CUDA toolkit. For optional local execution, choose documentation matching your installed tools. Links open online in a new tab; the questions and answer comparisons below work offline.
Use the documentation
Use the references to explain each answer, then apply it to this page’s example. No installation is needed.