Weight memory = params × bytes_per_param
Gradient memory = params × 2 bytes (BF16 gradients). LoRA reduces this by ~90%.
Optimizer memory = params × bytes_per_param × multiplier (AdamW = 2x, SGD = 0x)
Activation memory = batch × seq_len × hidden × layers × 2 bytes. Gradient checkpointing reduces by ~90%.
Temporary + overhead = ~2GB for framework buffers + ~2GB overhead
All values in GB. This is an estimate; actual usage varies by framework and GPU.