Gradient Descent Algorithm Teaching

via “gradient descent optimization with early exaggeration”

* 🏆 2009: [ImageNet: A large-scale hierarchical image database (ImageNet)](https://ieeexplore.ieee.org/document/5206848)

Unique: Two-phase optimization with early exaggeration (4x P scaling) specifically designed to overcome crowding problem and poor initialization; momentum scheduling (0.5 → 0.8) balances exploration and exploitation phases

vs others: More stable convergence than vanilla SGD; early exaggeration phase prevents collapse to trivial solutions that plague PCA-based initialization

via “gradient-descent-and-optimization-algorithm-comparison”


Unique: Animates parameter updates on loss landscapes to show how different optimizers navigate the optimization space, making algorithmic differences visible rather than abstract. Videos compare optimizers side-by-side showing convergence speed, stability, and final solution quality.

vs others: More intuitive than mathematical derivations, and more comprehensive than brief mentions in general ML courses

via “gradient-descent-algorithm-teaching”