Reference — LLM Toolkit

KEY FORMULAS

Memory — Weights

weights = params × bytes_per_param

BF16 = 2 bytes, FP32 = 4 bytes

Memory — Optimizer (AdamW)

optimizer = params × 2 × bytes_per_param

m and v buffers, each same size as weights

Memory — Activations

acts = batch × seq × hidden × layers × 2

BF16 activations. Checkpointing reduces ~90%

Perplexity

PPL = exp(cross_entropy_loss)

Lower is better. PPL=10 means 10^10 possible next tokens effectively

MFU

MFU = actual_FLOPs / peak_FLOPs

Target: 40-60% for training. Measures hardware efficiency.

KV Cache Memory

KV = 2 × seq × layers × kv_heads × head_dim × 2

BF16. GQA reduces by sharing KV heads.

USEFUL COMMANDS

nvidia-smi                    # GPU status
nvidia-smi dmon -s u          # Real-time utilization
nvidia-smi pmon -g all        # Per-process memory
nccl-tests/build/all_reduce_perf -b 8 -f 2 -g 1  # NCCL benchmark

with torch.profiler.profile() as prof:
    # training step
print(prof.key_averages().table(sort_by="cuda_time_total"))

# Single node, 4 GPUs
torchrun --nproc_per_node=4 train.py

# Multi-node
torchrun --nnodes=2 --nproc_per_node=8 \
  --node_rank=$RANK --master_addr=$ADDR \
  train.py

import torch
print(f"Allocated: {torch.cuda.memory_allocated()/1e9:.1f}GB")
print(f"Cached:    {torch.cuda.memory_reserved()/1e9:.1f}GB")
torch.cuda.max_memory_allocated()  # Peak memory

PRECISION REFERENCE

Format	Bits	Max Value	Training Use	Inference Use
FP32	32	~3.4×10³⁸	Accumulation only	Rarely
TF32	19 (truncated)	~6.8×10³⁸	Ampere+ matmul	-
BF16	16	~65504	Preferred	Common
FP16	16	~65504	Use with care	Common
FP8	8	~448 (E4M3)	Experimental	Hopper+ inference
INT8	8	127	-	Widespread
INT4	4	7	-	Aggressive quant

Verification

Specifications should be verified against current vendor documentation. Hardware capabilities change with driver and framework updates.