GPU Setup & Cloud

#Learning Objectives

Verify local GPU availability using nvidia-smi and PyTorch's CUDA API
Configure Google Colab with a T4 GPU for free cloud-based experiments
Benchmark matrix multiplication on CPU vs GPU and measure the speedup
Estimate the largest model that fits in your VRAM using the fp16 rule of thumb

#The Problem

Most lessons in phases 1-3 run fine on CPU. But once you start training CNNs, transformers, or LLMs (phases 4+), you need GPU acceleration. A training run that takes 8 hours on CPU takes 10 minutes on GPU.

You have three options: local GPU, cloud GPU, or Google Colab (free).

#The Concept

text

Your options:

1. Local NVIDIA GPU
   Cost: $0 (you already have it)
   Setup: Install CUDA + cuDNN
   Best for: Regular use, large datasets

2. Google Colab (free tier)
   Cost: $0
   Setup: None
   Best for: Quick experiments, no GPU at home

3. Cloud GPU (Lambda, RunPod, Vast.ai)
   Cost: $0.20-2.00/hr
   Setup: SSH + install
   Best for: Serious training, large models

#Build It

#Option 1: Local NVIDIA GPU

Check if you have one:

bash

nvidia-smi

Install PyTorch with CUDA:

python

import torch

print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")
if torch.cuda.is_available():
    print(f"GPU: {torch.cuda.get_device_name(0)}")
    print(f"Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")

#Option 2: Google Colab

Go to colab.research.google.com
Runtime > Change runtime type > T4 GPU
Run !nvidia-smi to verify

Upload notebooks from this course directly to Colab.

#Option 3: Cloud GPU

For Lambda Labs, RunPod, or Vast.ai:

bash

ssh user@your-gpu-instance

pip install torch torchvision torchaudio
python -c "import torch; print(torch.cuda.get_device_name(0))"

#No GPU? No problem.

Most lessons work on CPU. The ones that need GPU will say so and include Colab links.

python

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using: {device}")

#Build It: GPU vs CPU benchmark

python

import torch
import time

size = 5000

a_cpu = torch.randn(size, size)
b_cpu = torch.randn(size, size)

start = time.time()
c_cpu = a_cpu @ b_cpu
cpu_time = time.time() - start
print(f"CPU: {cpu_time:.3f}s")

if torch.cuda.is_available():
    a_gpu = a_cpu.to("cuda")
    b_gpu = b_cpu.to("cuda")

    torch.cuda.synchronize()
    start = time.time()
    c_gpu = a_gpu @ b_gpu
    torch.cuda.synchronize()
    gpu_time = time.time() - start
    print(f"GPU: {gpu_time:.3f}s")
    print(f"Speedup: {cpu_time / gpu_time:.0f}x")

#Exercises

Run the benchmark above and compare CPU vs GPU times
If you don't have a GPU, run it on Google Colab and compare
Check how much GPU memory you have and estimate the largest model you can fit (rule of thumb: 2 bytes per parameter for fp16)

#Key Terms

Term	What people say	What it actually means
CUDA	"GPU programming"	NVIDIA's parallel computing platform that lets you run code on the GPU
VRAM	"GPU memory"	Video RAM on the GPU, separate from system RAM. Limits model size.
fp16	"Half precision"	16-bit floating point, uses half the memory of fp32 with minimal accuracy loss
Tensor Core	"Fast matrix hardware"	Specialized GPU cores for matrix multiplication, 4-8x faster than regular cores

Run this lesson in the Code Lab

A real JupyterLab, with this lesson's code files (and a ready-to-run notebook). The Quick Lab opens instantly in your browser. The Cloud Lab adds a terminal, saved files and the full toolchain.

Quick Lab

python3 gpu_check.py

gpu_check.py

import time
import sys


def check_gpu():
    try:
        import torch
    except ImportError:
        print("PyTorch not installed. Run: pip install torch")
        return

    print("=== GPU Check ===\n")
    print(f"PyTorch version: {torch.__version__}")
    print(f"CUDA available: {torch.cuda.is_available()}")

    if not torch.cuda.is_available():
        print("\nNo GPU detected. That's fine for most lessons.")
        print("For GPU-heavy lessons, use Google Colab (free).")
        return

    print(f"CUDA version: {torch.version.cuda}")
    print(f"GPU: {torch.cuda.get_device_name(0)}")

    props = torch.cuda.get_device_properties(0)
    print(f"Memory: {props.total_memory / 1e9:.1f} GB")
    print(f"Compute capability: {props.major}.{props.minor}")

    print("\n=== CPU vs GPU Benchmark ===\n")
    size = 4000

    a = torch.randn(size, size)
    b = torch.randn(size, size)

    start = time.time()
    _ = a @ b
    cpu_time = time.time() - start
    print(f"CPU matrix multiply ({size}x{size}): {cpu_time:.3f}s")

    a_gpu = a.to("cuda")
    b_gpu = b.to("cuda")
    torch.cuda.synchronize()

    start = time.time()
    _ = a_gpu @ b_gpu
    torch.cuda.synchronize()
    gpu_time = time.time() - start
    print(f"GPU matrix multiply ({size}x{size}): {gpu_time:.3f}s")
    print(f"Speedup: {cpu_time / gpu_time:.0f}x")

    vram_gb = props.total_memory / 1e9
    params_fp16 = vram_gb * 1e9 / 2
    params_billions = params_fp16 / 1e9
    print(f"\nEstimated max model size (fp16): ~{params_billions:.0f}B parameters")


if __name__ == "__main__":
    check_gpu()

Checkpoint

5 questions on this lesson

Answer each one and see straight away whether you were right and why. Your first answer to a question counts. You are not signed in, so this is practice and nothing is saved.

Video coming soon

#Learning Objectives

#The Problem

#The Concept

#Build It

#Option 1: Local NVIDIA GPU

#Option 2: Google Colab

#Option 3: Cloud GPU

#No GPU? No problem.

#Build It: GPU vs CPU benchmark

#Exercises

#Key Terms

Run this lesson in the Code Lab

5 questions on this lesson