Accelerating Gradient Descent - Nesterov's Method

Gradient descent gets used to solve a myriad of machine learning problems - fast svm, regularized regression, reduced rank matrix approximation, deep belief networks, etc.  Nesterov's method can be used to speed up convergence for any of these.  It's simple to apply.  We'll run through some simple code that illustrates why gradient descent can sometimes be slow and then demonstrates the speedup with Nesterov's method.  Warning - this will be a little nerdy but extremely useful if your algo is too slow or won't converge.


Some References:

princeton - main theorem

princeton - background and needed lemma.

Derivation for more general function - (Tseng 2008)
google - On Accelerated Proximal Gradient Methods for Convex-Concave

In slide form (UCLA EE236)
google - 7. Fast proximal gradient methods

Join or login to comment.

  • Stephen

    For those of us on the waitlist, this is booked out?

    June 17, 2013

  • Charis K.

    any change of a web broadcast?

    June 15, 2013

10 went

Our Sponsors

People in this
Meetup are also in:

Create your own Meetup Group

Get started Learn more

Meetup has allowed me to meet people I wouldn't have met naturally - they're totally different than me.

Allison, started Women's Adventure Travel

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy