Weight memory = params × stored bytes (BF16/FP16 = 2, FP32 = 4, 4-bit = 0.5)
FP32 master weights = params × 4 bytes, counted for full training in BF16/FP16 (mixed precision keeps an FP32 copy; pure-FP32 needs none; LoRA freezes the base so none)
Gradient memory = trainable params × working bytes. Full training: all params. LoRA: adapters only (~1% of params — assumption; actual share depends on rank and target modules)
Optimizer memory = trainable params × 8 bytes for Adam/AdamW (FP32 m+v) or × 4 bytes for SGD with momentum. LoRA pays this on adapters only
Activation memory ≈ effective batch × seq_len × hidden × layers × bytes — a rough lower bound (attention backend and saved tensors shift it). Gradient checkpointing ≈ ×0.1 at ~20–30% extra compute
Temporary + overhead = ~2GB framework buffers + ~2GB overhead (CUDA context, fragmentation allowance)
Per-GPU = total ÷ GPU count (assumes replication; FSDP/ZeRO-3 shards weights+grads+optimizer ≈ ÷N)
Theoretical estimate, not a guarantee of fit. Verify with torch.cuda.max_memory_allocated() and keep ~20% headroom.