miniblog.

← Back to all posts

Wilfred Hughes

May 4, 2019 at 12:11

An excellent overview of the different gradient descent algorithms, and a nice example of content that is available as both a responsive website and a PDF on arXiv: https://ruder.io/optimizing-gradient-descent/

An overview of gradient descent optimization algorithms

An overview of gradient descent optimization algorithms

Gradient descent is the preferred way to optimize neural networks and many other machine learning algorithms but is often used as a black box. This post explores how many of the most popular gradient-based optimization algorithms such as Momentum, Adagrad, and Adam actually work.