Core Theory

• 1Backpropagation derivation: the chain rule applied to a computational graph. For a composed function f(g(x)), df/dx = (df/dg)(dg/dx). In a neural network, we compute gradients by propagating the upstream gradient backward through each operation. For linear layer y=Wx+b with upstream gradient δ=dL/dy: dL/dW = δ^T x (outer product), dL/dx = W^T δ, dL/db = δ. The backward pass is O(same) as the forward pass in FLOPs.
• 2Initialization theory: the goal is to start with weights such that the signal neither vanishes nor explodes through many layers. Xavier/Glorot initialization: Var(W) = 2/(fan_in + fan_out). Derived by requiring Var(output) = Var(input) for a linear activation. He initialization: Var(W) = 2/fan_in. Derived for ReLU (which zeros out half the activations on average, halving the effective fan_in). Bad initialization → vanishing gradients (weights too small) or exploding activations (weights too large) → slow or failed training.
• 3Universal Approximation Theorem: a single hidden layer neural network with sufficient width can approximate any continuous function on a compact set to arbitrary accuracy (Cybenko 1989 for sigmoid, Hornik 1991 for general activations). What it does NOT say: (1) how wide you need (can be exponential), (2) that you can find these weights by gradient descent, (3) that it will generalize. The theorem is about expressiveness, not learnability or generalization.
• 4Vanishing/exploding gradients in deep networks: in a deep network, gradients are products of many Jacobians. If ||W_i|| < 1 for each layer, gradients vanish exponentially with depth. If ||W_i|| > 1, they explode. ReLU helps (gradient is 1 or 0, not < 1). Residual connections (ResNets) fix this by providing a gradient highway: dL/dx = dL/d(x + F(x)) = dL/dx + dL/dF(x). The identity path ensures gradient magnitude never drops below 1.
• 5BatchNorm derivation and training vs. inference: forward: BN(x) = (x - μ_B) / √(σ²_B + ε) × γ + β, where μ_B and σ²_B are computed over the batch. Backward: the gradient must account for the fact that μ_B and σ²_B depend on x. During inference: use exponential moving averages of μ and σ² from training (the 'running statistics'). This different behavior between train and eval is the source of many production bugs.
• 6Double descent phenomenon: classical bias-variance predicts a U-shaped test error as model size increases (too simple = underfitting, too complex = overfitting). Double descent shows a second descent for very large models: test error peaks at the interpolation threshold (where the model exactly fits training data) then decreases again in the overparameterized regime. This explains why making neural networks larger often improves generalization even without regularization.
• 7Neural Tangent Kernel (NTK): in the infinite-width limit, a neural network trained with gradient descent behaves like kernel regression with a fixed kernel (the NTK). This provides theoretical understanding of why neural networks generalize and why they can be trained with simple gradient descent. Practical implication: wide networks are more linear and easier to train, narrow networks are more nonlinear but harder to optimize. The NTK theory is an active research area explaining neural network behavior.
• 8Pre-norm vs. post-norm in transformers: pre-norm (LN before attention/FFN, residual around the whole block) vs. post-norm (LN after the residual addition). Empirically, pre-norm trains more stably for deep transformers because gradients flow through the residual connection without passing through the LN operation, which can distort gradient magnitudes. GPT-2, GPT-3, LLaMA all use pre-norm. The original Transformer paper used post-norm.