Vision Transformer, Part 6 — How It Learns: Logits → Loss → Backprop → Update

2026-07-26 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Advanced, medium

Summary

Vision Transformer, Part 6" elucidates the backward pass of a ViT's training, detailing how a single loss value updates its 1.2 million weights. The process begins by converting 10 logits to probabilities via softmax, then calculating cross-entropy loss against a one-hot label, resulting in a loss of 0.13 for a horse image with 0.88 probability. This loss, averaged over a batch of 128 images, initiates backpropagation. The gradient at each logit is `p − y` (e.g., `0.88 − 1 = −0.12` for the horse logit), which propagates backward as `δ`. Each weight's gradient is `δ · x`, with the chain rule determining its direction. The AdamW optimizer refines updates with momentum, per-parameter rates, and decoupled weight decay (0.05), using a 3e-4 learning rate and 1.0 gradient clip. This loop runs ~19,500 times, reducing loss from ~2.3 to ~0.8 and achieving 82.68% validation accuracy.

Key takeaway

For machine learning engineers optimizing Vision Transformer training, understanding the backward pass mechanics is crucial. You should meticulously configure your loss function (e.g., cross-entropy with label smoothing 0.1) and AdamW optimizer settings (lr 3e-4, weight decay 0.05, gradient clip 1.0). This precise control over how gradients propagate and weights update directly impacts model convergence, loss reduction from ~2.3 to ~0.8, and achieving target accuracy like 82.68%.

Key insights

The entire forward pass must be differentiable for a single loss to adjust all model weights via backpropagation.

Principles

• Every operation from pixels to loss must be differentiable.
• Gradient descent automatically reduces loss by stepping opposite the gradient.
• AdamW refines updates with momentum and per-parameter rates.

Method

The learning process involves logits → softmax → cross-entropy loss → backpropagation (gradient `p − y` then `δ · x`) → chain rule for direction → AdamW weight update (momentum, per-parameter rate, weight decay).

In practice

• Implement cross-entropy loss with label smoothing.
• Use AdamW optimizer with cosine-decayed learning rate.
• Monitor loss reduction and validation accuracy over epochs.

Topics

• Vision Transformer
• Backpropagation
• Cross-Entropy Loss
• AdamW Optimizer
• Gradient Descent
• Neural Network Training

Best for: AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Optimizers in Deep Learning: From Gradient Descent to Adam — Optimizers guide neural network training by efficiently updating parameters to minimize loss, evolving from basic GD to adaptive Adam.
• Backpropagation: Computational Graph Derivation — The algorithm behind modern deep learning — Backpropagation efficiently computes gradients for deep learning models by recursively applying the chain rule through computational graphs.
• Distribution-Free Pretraining of Classification Losses via Evolutionary Dynamics — EDL pretrains transferable classification losses using synthetic data and evolutionary optimization with chaotic mutation.
• CNN Architecture Evolution: ResNet → EfficientNet → ConvNeXt — What Actually Changed? — CNN performance gains are equally driven by architectural innovation and advanced training recipes.
• Linearizing Vision Transformer with Test-Time Training — Linearizing Vision Transformers with Test-Time Training enables efficient weight transfer and faster inference.
• Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate — Maximizing the embedding layer learning rate is crucial for effective hyperparameter transfer in large language model training.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.