Research Scientist
Computer Vision
Computer Vision spans from low-level feature learning (CNNs) to high-level scene understanding (detection, segmentation) and modern representation learning (self-supervised ViTs). Research Scientists at Google DeepMind, Meta FAIR, and Apple Vision Pro teams are expected to understand architectural innovations deeply โ why ResNets work, how ViT handles variable-length inputs, how anchor-free detection eliminates design choices โ as well as evaluation metrics (mAP, IoU, FID) and the shift from supervised to self-supervised pre-training.
Core Theory
- 1CNN fundamentals โ receptive field: the receptive field (RF) of a feature map location is the region of the original input that can influence that location. For a stack of conv layers each with kernel k and stride s: RF grows as RF_{l+1} = RF_l + (k-1) ร โ_{i<l} s_i. With k=3, s=1: RF grows by 2 per layer, so a 10-layer network has RF = 21. With stride-2 layers: RF grows much faster. Dilated (atrous) convolutions expand RF without increasing parameters: a 3ร3 conv with dilation d has effective kernel size d(k-1)+1, RF contribution = 2d. This is the key insight behind DeepLab โ using large effective RF for segmentation without downsampling spatially.
- 2ResNet architecture: the key insight is the residual (skip) connection: x_{l+1} = F(x_l, W_l) + x_l. Each block learns a residual F(x) = 0 is easier than learning the identity mapping y = x from scratch. Bottleneck blocks (1ร1 โ 3ร3 โ 1ร1) reduce computation by using 1ร1 convolutions to compress channels before the expensive 3ร3 conv. ResNet-50 uses 4 stages with [3,4,6,3] bottleneck blocks. Gradient highway: โL/โx_l = โL/โx_{l+1} (I + โF/โx_l), so gradients never vanish through skip connections. DenseNet extends this by connecting every layer to every subsequent layer: x_l = H_l([x_0, x_1, ..., x_{l-1}]), enabling feature reuse at O(Lยฒ) connections.
- 3EfficientNet โ compound scaling: instead of scaling width, depth, or resolution independently, EfficientNet jointly scales all three with a fixed ratio. Given a resource constraint ฯ, scale: depth d = ฮฑ^ฯ, width w = ฮฒ^ฯ, resolution r = ฮณ^ฯ, with ฮฑ ร ฮฒยฒ ร ฮณยฒ โ 2 (FLOP constraint). The optimal ฮฑ, ฮฒ, ฮณ are found by Neural Architecture Search on a small base model. Compound scaling works because larger input โ needs deeper network to increase RF; needs wider network to capture more patterns. Separately scaling any one dimension has diminishing returns. EfficientNet-B7 achieves top-1 accuracy 84.4% on ImageNet with 8ร fewer parameters than other models at comparable accuracy.
- 4Vision Transformer (ViT): images are split into fixed-size patches (e.g., 16ร16), each linearly projected to a D-dimensional embedding. A special [CLS] token is prepended. 1D learnable positional embeddings are added. Standard Transformer encoder layers (multi-head self-attention + FFN) process the patch sequence. The [CLS] token output is used for classification. Key differences from NLP: (1) 2D spatial structure โ ViT ignores it (1D positional embedding), but 2D RPE or sinusoidal 2D encodings help for detection/segmentation; (2) data hunger โ ViT needs large datasets (JFT-300M) or strong augmentation to match CNNs; (3) no built-in inductive bias (translation equivariance) โ must be learned from data, requiring more training data than CNNs.
- 5DETR โ end-to-end object detection with Transformers: takes a CNN backbone + positional encoding, passes features through a Transformer encoder-decoder. The decoder uses N learned 'object queries' that attend to encoder features via cross-attention. Each query decodes to a (class, bounding box) pair. Training uses Hungarian matching: find the optimal assignment between predictions and ground-truth boxes (bipartite matching with cost = class loss + L1 box loss + GIoU loss), then compute the matched cross-entropy + regression loss. No NMS, no anchors, no heuristics. DETR's attention maps show that object queries specialize to different regions/scales โ empirically, some queries attend to small objects, others to large. Limitation: slow convergence (500 epochs) due to the decoder needing to learn when to 'stop' searching.
- 6Object Detection โ YOLO to YOLOv8: YOLO ('You Only Look Once') divides the image into an SรS grid. Each cell predicts B bounding boxes (x,y,w,h,confidence) and C class probabilities. YOLOv1 predicts 2 boxes per cell, 20 classes. YOLOv2 introduced anchor boxes: pre-defined aspect ratios clustered from training data, predictions are offsets from anchors. YOLOv3 uses multi-scale prediction at 3 FPN scales (for small, medium, large objects). Faster R-CNN uses a Region Proposal Network (RPN) that slides over feature maps, predicting objectness + box deltas for each anchor, then RoI pooling extracts fixed-size features for each proposal. FCOS eliminates anchors entirely: each point on the feature map directly predicts (l,r,t,b) distances to the bounding box edges, plus centerness = โ(min(l,r)/max(l,r) ร min(t,b)/max(t,b)) to suppress low-quality predictions.
- 7Semantic Segmentation โ from FCN to DeepLab: FCN (Fully Convolutional Network) replaces the fully-connected layers of a classifier with 1ร1 convolutions, outputting a spatial prediction map. Skip connections from intermediate feature maps recover spatial detail. U-Net extends this: encoder (downsampling with max-pool) + decoder (upsampling) with skip connections concatenated at each scale. Skip connections pass high-resolution encoder features to the decoder, preserving fine boundaries. DeepLabv3+ uses dilated convolutions to maintain large RF without downsampling, plus ASPP (Atrous Spatial Pyramid Pooling): multiple parallel dilated convolutions with different dilation rates (1, 6, 12, 18) capture multi-scale context. The ASPP outputs are concatenated and fused with a shallow 1/4-resolution encoder stream for boundary refinement.
- 8Self-supervised learning โ SimCLR: learns representations by maximizing agreement between two augmented views of the same image. Two augmented crops x_i, x_j from the same image โ encoders f (shared weights) โ projectors g โ embeddings z_i, z_j. NT-Xent (InfoNCE) loss: โ(i,j) = -log[exp(sim(z_i,z_j)/ฯ) / ฮฃ_{kโ i} exp(sim(z_i,z_k)/ฯ)] where sim is cosine similarity. The denominator sums over all 2N-2 other samples in the batch as negatives. Critical tricks: (1) large batch size (8192) for more negatives; (2) strong augmentation (crop + color jitter + grayscale + blur); (3) projector head g (2-layer MLP) discards augmentation-invariant information. The representation before g is used for downstream tasks.
- 9Self-supervised learning โ DINO (Distillation with No Labels): uses a student-teacher framework with no contrastive negatives. Student and teacher share architecture (ViT); teacher uses exponential moving average (EMA) of student weights (no gradient). Same image is augmented into multiple crops: teacher sees global crops, student sees local crops + global crops. Loss: cross-entropy between student and teacher softmax outputs (with centering to prevent collapse). DINO-ViT attention heads learn to segment objects (without segmentation supervision!) because the [CLS] token attends to semantically relevant patches. DINOv2 trains on a curated 142M image dataset with distillation, achieving SOTA on many dense prediction tasks.
- 10MAE (Masked Autoencoder): asymmetric autoencoder where (1) encoder processes only the 25% of patches that are NOT masked โ no mask tokens in encoder โ significant compute saving; (2) decoder reconstructs masked patches from encoded visible tokens + lightweight learnable mask tokens. Loss: MSE on pixel values of masked patches only. Key insight: images are highly redundant โ masking 75% still leaves enough context to reconstruct masked patches. This forces the encoder to learn holistic image representations rather than low-level texture. Compared to SimCLR/DINO: MAE does not need negative pairs or multiple crops, scales better with model size, and the decoder is cheap (8 Transformer blocks vs 24 in ViT-L encoder). MAE linear probe slightly underperforms DINO, but fine-tuning performance is competitive or better.
Key Formulas
| 1 | import torch |
| 2 | import torch.nn as nn |
| 3 | import torch.nn.functional as F |
| 4 | import math |
| 5 | |
| 6 | # โโโ Vision Transformer Patch Embedding โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 7 | |
| 8 | class PatchEmbedding(nn.Module): |
| 9 | """ |
| 10 | Split image into non-overlapping patches and project to d_model. |
| 11 | Image: (B, C, H, W) โ patches: (B, N, d_model) |
| 12 | where N = (H/P) * (W/P), P = patch size |
| 13 | """ |
| 14 | def __init__(self, img_size=224, patch_size=16, in_channels=3, d_model=768): |
| 15 | super().__init__() |
| 16 | assert img_size % patch_size == 0, "Image size must be divisible by patch size" |
| 17 | self.num_patches = (img_size // patch_size) ** 2 |
| 18 | self.patch_size = patch_size |
| 19 | # Use a single Conv2d with kernel=patch_size, stride=patch_size |
| 20 | # This efficiently extracts and projects patches in one step |
| 21 | self.proj = nn.Conv2d(in_channels, d_model, |
| 22 | kernel_size=patch_size, stride=patch_size) |
| 23 | |
| 24 | def forward(self, x): |
| 25 | # x: (B, C, H, W) |
| 26 | x = self.proj(x) # (B, d_model, H/P, W/P) |
| 27 | x = x.flatten(2) # (B, d_model, N) |
| 28 | x = x.transpose(1, 2) # (B, N, d_model) โ standard sequence format |
| 29 | return x |
| 30 | |
| 31 | |
| 32 | # โโโ Multi-Head Self-Attention โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 33 | |
| 34 | class MultiHeadSelfAttention(nn.Module): |
| 35 | """ |
| 36 | Scaled dot-product attention: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V |
| 37 | Multi-head: run h attention heads in parallel, concatenate and project. |
| 38 | """ |
| 39 | def __init__(self, d_model: int, n_heads: int, dropout: float = 0.0): |
| 40 | super().__init__() |
| 41 | assert d_model % n_heads == 0 |
| 42 | self.n_heads = n_heads |
| 43 | self.d_k = d_model // n_heads |
| 44 | |
| 45 | self.qkv_proj = nn.Linear(d_model, 3 * d_model) # fused Q, K, V projection |
| 46 | self.out_proj = nn.Linear(d_model, d_model) |
| 47 | self.dropout = nn.Dropout(dropout) |
| 48 | |
| 49 | def forward(self, x, return_attention: bool = False): |
| 50 | B, N, D = x.shape |
| 51 | |
| 52 | # Project and split into Q, K, V |
| 53 | qkv = self.qkv_proj(x) # (B, N, 3*D) |
| 54 | qkv = qkv.reshape(B, N, 3, self.n_heads, self.d_k) |
| 55 | qkv = qkv.permute(2, 0, 3, 1, 4) # (3, B, n_heads, N, d_k) |
| 56 | Q, K, V = qkv[0], qkv[1], qkv[2] # each (B, n_heads, N, d_k) |
| 57 | |
| 58 | # Scaled dot-product attention |
| 59 | scale = math.sqrt(self.d_k) |
| 60 | attn_logits = torch.matmul(Q, K.transpose(-2, -1)) / scale # (B, h, N, N) |
| 61 | attn_weights = F.softmax(attn_logits, dim=-1) |
| 62 | attn_weights = self.dropout(attn_weights) |
| 63 | |
| 64 | # Apply attention to values |
| 65 | out = torch.matmul(attn_weights, V) # (B, h, N, d_k) |
| 66 | out = out.transpose(1, 2).reshape(B, N, D) # (B, N, D) โ concat heads |
| 67 | out = self.out_proj(out) |
| 68 | |
| 69 | if return_attention: |
| 70 | return out, attn_weights # useful for DINO visualization |
| 71 | return out |
| 72 | |
| 73 | |
| 74 | # โโโ ViT Block โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 75 | |
| 76 | class ViTBlock(nn.Module): |
| 77 | """Standard Transformer block with pre-norm (used in GPT-2, LLaMA, ViT variants).""" |
| 78 | def __init__(self, d_model: int, n_heads: int, mlp_ratio: float = 4.0, dropout: float = 0.0): |
| 79 | super().__init__() |
| 80 | self.norm1 = nn.LayerNorm(d_model) |
| 81 | self.attn = MultiHeadSelfAttention(d_model, n_heads, dropout) |
| 82 | self.norm2 = nn.LayerNorm(d_model) |
| 83 | # FFN: expand to mlp_ratio*d_model, GELU, contract back |
| 84 | mlp_dim = int(d_model * mlp_ratio) |
| 85 | self.mlp = nn.Sequential( |
| 86 | nn.Linear(d_model, mlp_dim), nn.GELU(), |
| 87 | nn.Dropout(dropout), |
| 88 | nn.Linear(mlp_dim, d_model), nn.Dropout(dropout), |
| 89 | ) |
| 90 | |
| 91 | def forward(self, x): |
| 92 | # Pre-norm variant: LN before attention, residual around the block |
| 93 | x = x + self.attn(self.norm1(x)) |
| 94 | x = x + self.mlp(self.norm2(x)) |
| 95 | return x |
| 96 | |
| 97 | |
| 98 | # โโโ Mini ViT for Image Classification โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 99 | |
| 100 | class ViT(nn.Module): |
| 101 | def __init__(self, img_size=224, patch_size=16, in_channels=3, |
| 102 | d_model=768, n_heads=12, n_layers=12, num_classes=1000, |
| 103 | dropout=0.1): |
| 104 | super().__init__() |
| 105 | self.patch_embed = PatchEmbedding(img_size, patch_size, in_channels, d_model) |
| 106 | num_patches = self.patch_embed.num_patches |
| 107 | |
| 108 | # Learnable [CLS] token โ prepended to patch sequence |
| 109 | self.cls_token = nn.Parameter(torch.zeros(1, 1, d_model)) |
| 110 | # Learnable 1D positional embedding (includes CLS position) |
| 111 | self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, d_model)) |
| 112 | nn.init.trunc_normal_(self.cls_token, std=0.02) |
| 113 | nn.init.trunc_normal_(self.pos_embed, std=0.02) |
| 114 | |
| 115 | self.dropout = nn.Dropout(dropout) |
| 116 | self.blocks = nn.Sequential(*[ViTBlock(d_model, n_heads, dropout=dropout) |
| 117 | for _ in range(n_layers)]) |
| 118 | self.norm = nn.LayerNorm(d_model) |
| 119 | self.head = nn.Linear(d_model, num_classes) |
| 120 | |
| 121 | def forward(self, x): |
| 122 | B = x.shape[0] |
| 123 | x = self.patch_embed(x) # (B, N, D) |
| 124 | |
| 125 | # Prepend CLS token |
| 126 | cls = self.cls_token.expand(B, -1, -1) # (B, 1, D) |
| 127 | x = torch.cat([cls, x], dim=1) # (B, N+1, D) |
| 128 | |
| 129 | # Add positional embedding |
| 130 | x = x + self.pos_embed |
| 131 | x = self.dropout(x) |
| 132 | |
| 133 | # Transformer blocks |
| 134 | x = self.blocks(x) |
| 135 | x = self.norm(x) |
| 136 | |
| 137 | # Use CLS token for classification |
| 138 | cls_out = x[:, 0] # (B, D) |
| 139 | return self.head(cls_out) # (B, num_classes) |
| 140 | |
| 141 | |
| 142 | # โโโ SimCLR NT-Xent Loss โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 143 | |
| 144 | def nt_xent_loss(z1: torch.Tensor, z2: torch.Tensor, temperature: float = 0.07): |
| 145 | """ |
| 146 | NT-Xent (Normalized Temperature-scaled Cross Entropy) loss for SimCLR. |
| 147 | |
| 148 | z1, z2: (N, D) โ two views of the same N images |
| 149 | Positive pair: (z1[i], z2[i]) โ same image, different augmentations |
| 150 | Negatives: all other 2N-2 samples in the batch |
| 151 | |
| 152 | Implementation: concatenate z1 and z2 โ (2N, D), compute all-pairs similarity, |
| 153 | mask out diagonal, use NT-Xent loss. |
| 154 | """ |
| 155 | N = z1.shape[0] |
| 156 | |
| 157 | # L2 normalize embeddings (cosine similarity = dot product after normalization) |
| 158 | z1 = F.normalize(z1, dim=-1) |
| 159 | z2 = F.normalize(z2, dim=-1) |
| 160 | |
| 161 | # Concatenate to form 2N embeddings: [z1_0,...,z1_N, z2_0,...,z2_N] |
| 162 | z = torch.cat([z1, z2], dim=0) # (2N, D) |
| 163 | |
| 164 | # Compute all pairwise cosine similarities |
| 165 | sim = torch.mm(z, z.T) / temperature # (2N, 2N) |
| 166 | |
| 167 | # Mask out self-similarity (diagonal) |
| 168 | mask = torch.eye(2 * N, dtype=torch.bool, device=z.device) |
| 169 | sim = sim.masked_fill(mask, -1e9) |
| 170 | |
| 171 | # Positive pairs: (i, i+N) and (i+N, i) for i in 0..N-1 |
| 172 | # Labels: the positive index for sample i is i+N (and vice versa) |
| 173 | labels = torch.cat([torch.arange(N, 2*N), torch.arange(0, N)]).to(z.device) |
| 174 | |
| 175 | loss = F.cross_entropy(sim, labels) |
| 176 | return loss |
| 177 | |
| 178 | |
| 179 | # โโโ Demo โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ |
| 180 | |
| 181 | if __name__ == "__main__": |
| 182 | torch.manual_seed(42) |
| 183 | |
| 184 | # Test ViT-Tiny (fewer layers/heads for demo) |
| 185 | vit = ViT(img_size=224, patch_size=16, d_model=192, n_heads=3, n_layers=4, num_classes=100) |
| 186 | x = torch.randn(4, 3, 224, 224) # batch of 4 images |
| 187 | logits = vit(x) |
| 188 | print(f"ViT output shape: {logits.shape}") # (4, 100) |
| 189 | print(f"Num patches: {vit.patch_embed.num_patches}") # 196 = (224/16)^2 |
| 190 | |
| 191 | # Test SimCLR loss |
| 192 | z1 = torch.randn(32, 128) # 32 images, 128-dim projections |
| 193 | z2 = torch.randn(32, 128) # second views |
| 194 | loss = nt_xent_loss(z1, z2, temperature=0.07) |
| 195 | print(f"NT-Xent loss: {loss.item():.4f}") # ~log(63) โ 4.14 for random init |
Three core CV components: (1) PatchEmbedding uses a single Conv2d with stride=patch_size to efficiently extract and project image patches โ equivalent to manual patch extraction + linear projection but faster. (2) MultiHeadSelfAttention uses a fused QKV projection for efficiency and returns optional attention weights (used in DINO to visualize which patches the CLS token attends to). (3) NT-Xent loss concatenates both views into a 2N batch, computes all pairwise similarities, and uses cross-entropy with the correct positive pair as the target class โ the clean vectorized implementation that handles both directions of the contrastive loss simultaneously.
Worked Interview Problems
3 problemsProblem
VGG-16 has: 2ร Conv(3ร3,s=1), MaxPool(2ร2,s=2), 2ร Conv(3ร3,s=1), MaxPool(2ร2,s=2), 3ร Conv(3ร3,s=1), MaxPool(2ร2,s=2), 3ร Conv(3ร3,s=1), MaxPool(2ร2,s=2). Calculate the receptive field after each stage.
Solution
Formula: RF_l = RF_{l-1} + (k_l - 1) ร stride_product_{1..l-1}. Initialize RF=1, stride_product=1.
Stage 1 โ 2ร Conv(3ร3,s=1): RF after conv1: 1 + (3-1)ร1 = 3. RF after conv2: 3 + (3-1)ร1 = 5. MaxPool(s=2): RF after pool1 = 5 + (2-1)ร1 = 6. stride_product = 2.
Stage 2 โ 2ร Conv(3ร3,s=1): RF after conv3 = 6 + 2ร2 = 10. RF after conv4 = 10 + 2ร2 = 14. MaxPool: 14 + 1ร2 = 16. stride_product = 4.
Stage 3 โ 3ร Conv(3ร3,s=1): RF = 16 + 2ร4 = 24 โ 32 โ 40. MaxPool: 40 + 1ร4 = 44. stride_product = 8.
Stage 4 โ 3ร Conv(3ร3,s=1): RF = 44 + 2ร8 = 60 โ 76 โ 92. MaxPool: 92 + 1ร8 = 100. stride_product = 16.
Stage 5 โ 3ร Conv(3ร3,s=1): RF = 100 + 2ร16 = 132 โ 164 โ 196. MaxPool: 196 + 1ร16 = 212. Final RF โ 212 pixels for a 224ร224 input โ nearly the full image.
Answer
VGG-16 final receptive field โ 212 pixels (for 224ร224 input). The RF grows linearly per conv layer and jumps multiplicatively at each pooling layer. Deep networks achieve very large RFs, but neurons at the network's output 'see' almost the entire input image, which is why global context is captured without explicit global pooling.
Common Mistakes
Confusing mAP computation: mAP averages Average Precision over all classes. AP for one class is the area under the Precision-Recall curve (using either 11-point interpolation in PASCAL VOC or COCO's 101-point). COCO mAP averages over IoU thresholds [0.5:0.05:0.95] โ significantly harder than VOC's single IoU=0.5 threshold. When someone says 'mAP=50', you must ask: COCO or VOC? 50 COCO is world-class; 50 VOC is poor.
Thinking ViT has less inductive bias than CNN, therefore always better: ViT lacks translation equivariance and locality bias. On small datasets (<100K images) with standard augmentation, CNNs (ResNet, EfficientNet) typically outperform ViT. ViT excels at scale (large datasets + large model). DeiT showed that strong data augmentation + knowledge distillation closes the gap on ImageNet, but for truly small datasets, CNNs are still preferable.
Forgetting that DETR's N=100 queries are permutation-invariant during inference but trained with fixed-N matching: each query learns a different specialization through training (some focus on small objects in corners, others on large centered objects). This emergent specialization is not designed in โ it comes from the Hungarian matching consistently assigning the same types of objects to the same queries.
Mixing up dilated convolution's effect on feature map size vs. receptive field: a dilated conv with dilation d and kernel k does NOT change the output spatial size (same as stride-1 conv). It only increases the receptive field. Downsampling only happens with stride > 1 or pooling. DeepLab uses dilation to maintain high-resolution feature maps (output stride 8 or 16) while having large RF โ critical for segmentation.
Claiming SimCLR works with small batches: NT-Xent loss requires enough in-batch negatives to work. With N=64 samples and 2 views, you have only 126 negatives per sample. The discriminative task is too easy โ representations are poor. Practical minimum: Nโฅ512. MoCo solves this by maintaining a momentum-updated queue of negatives (size 65536) decoupled from batch size.