Core Theory

• 1Knowledge distillation — soft targets and temperature: train a small 'student' network to mimic a large 'teacher'. Hinton et al. 2015: instead of training on hard labels (one-hot), train on the teacher's softmax outputs ('soft targets'). Loss: L = α × L_CE(student, hard_labels) + (1-α) × T² × L_KL(student_T, teacher_T), where student_T = softmax(z_s/T), teacher_T = softmax(z_t/T), T > 1 is temperature. The T² factor compensates for the reduced gradient magnitude from soft targets. Why soft targets help: the teacher's distribution encodes 'dark knowledge' — e.g., P(cat)=0.01 for a dog image tells the student that cats are the most plausible mistake. At T=1 this signal is tiny; T=3-10 amplifies it. The student achieves near-teacher performance with 10-100× fewer parameters.
• 2Intermediate feature distillation and Born-Again Networks: beyond soft targets, distill intermediate representations. FitNets: student's hidden features should match teacher's via a regressor layer. AT (Attention Transfer): student's spatial attention maps should match teacher's. PKD (Patient Knowledge Distillation): match all transformer layers, not just the final one — patient KD is the technique used in DistilBERT. Born-Again Networks: train student with same architecture as teacher — surprisingly improves performance beyond a single model. The student has access to the teacher's exact posterior, which acts as a smooth, less-overfit training signal. Ensembling multiple generations of BANs further improves performance.
• 3Pruning — unstructured vs. structured: weight pruning removes individual weights (sparse tensors). Magnitude pruning: remove weights where |w| < threshold. Problem: unstructured sparsity does NOT speed up inference on standard hardware (GPUs compute dense matrix multiplications). Structured pruning removes entire neurons, attention heads, or layers → actual speedup. Lottery Ticket Hypothesis (Frankle & Carlin 2019): a random initialization contains a 'winning ticket' — a sparse subnetwork that, when trained from the same initialization (not retrained from random), matches the full network's accuracy. Finding the ticket: (1) train full network; (2) prune the p% smallest magnitude weights; (3) rewind weights to original initialization (not zero!); (4) retrain sparse network. Critically, reinitializing to random (not original init) fails — the init matters.
• 4Iterative magnitude pruning (IMP): pruning then retraining is more effective than pruning all at once. Algorithm: (1) train full network; (2) prune 20% of remaining weights by magnitude; (3) retrain/fine-tune; (4) repeat steps 2-3 until target sparsity. With IMP, models can reach 90%+ sparsity with minimal accuracy loss (lottery tickets exist at all scales). Practical issue: 90% sparse models on GPU are not faster without special sparse CUDA kernels. Hardware acceleration for sparsity: NVIDIA Ampere's 2:4 structured sparsity (exactly 2 non-zeros per 4-element group) gives 2× theoretical speedup and is hardware-accelerated.
• 5Post-training quantization (PTQ): convert a trained float32 model to int8 or int4 without retraining. Uniform affine quantization: Q(x) = round(x/s) - z where scale s = (x_max - x_min)/(2^b - 1), zero-point z corrects for asymmetric ranges. Quantization error: MSE(x, dequant(Q(x))) ≈ s²/12 (uniform quantization error is approximately uniform in [0, s²/12]). Problem: weight distributions in transformers have outlier channels with very large magnitudes → small s (to cover outliers) → all other weights map to same integer → catastrophic accuracy loss. LLM.int8() (Dettmers et al. 2022): decompose matrix multiplication into two parts — (1) outlier channels (columns with large magnitude) computed in float16, (2) remaining channels quantized to int8. Combines both results. This 2-step decomposition enables 8-bit inference for 175B+ parameter models.
• 6GPTQ — one-shot quantization for LLMs: uses the Optimal Brain Compression (OBC) framework. For each row of weight matrix W, minimize quantization error: argmin_{Q} ||W X - Q X||_F² where X is calibration data. Solves the optimal quantization order using the Hessian H = 2XX^T. GPTQ: quantize weights in order of Hessian-weighted sensitivity, updating remaining unquantized weights to compensate (lazy batch update). Achieves INT4 quantization with minimal accuracy loss on GPT models. Key insight: instead of independently quantizing each weight (which ignores correlations), GPTQ updates remaining weights to correct for quantization error — a second-order method.
• 7LoRA — Low-Rank Adaptation: for a pre-trained weight matrix W_0 ∈ R^{d×k}, constrain the update to be low-rank: W = W_0 + ΔW = W_0 + BA where B ∈ R^{d×r}, A ∈ R^{r×k}, r << min(d,k). Only A and B are trained; W_0 is frozen. Initialization: A ~ N(0, σ²), B = 0 → ΔW = 0 at init (no change to model behavior). FLOPs during forward: full W_0 forward + BA forward. No extra FLOPs at inference if merged: W = W_0 + BA (merge before deployment, zero inference overhead). Rank selection: r=4,8,16 covers most use cases. Theoretical justification: Aghajanyan et al. 2021 showed pre-trained models have very low intrinsic dimensionality — fine-tuning tasks live in a much lower-dimensional subspace than the full parameter space.
• 8QLoRA — quantized LoRA: combine 4-bit quantization (NF4 — Normal Float 4, optimal for normally distributed weights) with LoRA. W_0 stored in NF4 (4-bit); during forward pass, dequantize to BF16 for computation; LoRA adapters A, B stored in BF16. Key innovation: double quantization — quantize the quantization constants themselves (saves ~0.4 bits per parameter). Paged optimizer: use NVIDIA's unified memory to page optimizer states to CPU RAM when GPU memory is full, enabling fine-tuning 65B parameter models on a single A100 80GB. QLoRA enables fine-tuning LLaMA-65B on one GPU — previously requiring 8× A100s.
• 9Flash Attention — IO-aware exact attention: standard attention materializes the full n×n attention matrix in HBM (GPU DRAM), requiring O(n²) memory read/write operations. HBM bandwidth is the bottleneck (not FLOP throughput). Flash Attention (Dao et al. 2022): tile the Q, K, V matrices into blocks that fit in SRAM (GPU L1 cache). Compute attention block by block, using the online softmax algorithm (Milakov & Gimelshein 2018) to correctly normalize without seeing the full row. Never materialize the full n×n matrix in HBM — only write the final O output. Memory: O(n) instead of O(n²). Speed: 2-4× faster on A100 for typical sequence lengths (1K-8K). Asymptotically same FLOP count as standard attention — speedup is purely from fewer HBM read/writes. Flash Attention v2/v3 adds parallelism and further reduces non-matmul FLOPs.
• 10Speculative decoding: LLM inference is memory-bandwidth bound (loading 70B parameters every token). Idea: use a small 'draft' model (7B) to generate k=4-8 draft tokens quickly, then verify all k+1 tokens in parallel with the large model (using a single batched forward pass). Accept each draft token if P_large(token | context) / P_draft(token | context) > uniform random → guarantees the large model's token distribution is preserved exactly (rejection sampling). Average acceptance rate: 60-80% for good draft/target model pairs. Speedup: 2-3× on typical generation tasks (coding, conversation). Key insight: the large model does k+1 tokens in 1 forward pass cost (batched verification) instead of k+1 sequential passes. Works because language is locally predictable — draft model gets most tokens right.