Accelerating Gradient Descent - Nesterov's Method

Gradient descent gets used to solve a myriad of machine learning problems - fast svm, regularized regression, reduced rank matrix approximation, deep belief networks, etc.  Nesterov's method can be used to speed up convergence for any of these.  It's simple to apply.  We'll run through some simple code that illustrates why gradient descent can sometimes be slow and then demonstrates the speedup with Nesterov's method.  Warning - this will be a little nerdy but extremely useful if your algo is too slow or won't converge.

 

Some References:

princeton - main theorem
http://blogs.princeton.edu/imabandit/2013/04/01/acceleratedgradientdescent/

princeton - background and needed lemma.
http://blogs.princeton.edu/imabandit/2013/03/28/smoothfunctions/

Derivation for more general function - (Tseng 2008)
google - On Accelerated Proximal Gradient Methods for Convex-Concave

In slide form (UCLA EE236)
google - 7. Fast proximal gradient methods

Join or login to comment.

  • Stephen

    For those of us on the waitlist, this is booked out?

    June 17, 2013

  • Charis K.

    any change of a web broadcast?

    June 15, 2013

Our Sponsors

  • Amazon AWS

    Free compute time for working group events

  • Cloudera

    Access to Hadoop contributors

  • Safari Online Books

    Free subscription for Apache Bigtop Working Group members.

  • NVIDIA

    Technical support, meeting space, and access to GPUs

People in this
Meetup are also in:

Create a Meetup Group and meet new people

Get started Learn more
Allison

Meetup has allowed me to meet people I wouldn't have met naturally - they're totally different than me.

Allison, started Women's Adventure Travel

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy