Model Hyperparameter Tuning

Our next meetup will be a series of presentations on hyperparameter optimization (https://en.wikipedia.org/wiki/Hyperparameter_optimization) and how to use various software packages to find a set of optimal hyperparameters for your machine learning model.

Model selection via hyperparameter optimization is an important part of machine learning and we will discuss both the very basic and sophisticated methods for tuning models. Including:

1. Cartesian Grid Search (https://en.wikipedia.org/wiki/Hyperparameter_optimization#Grid_search)
2. Random Grid Search (https://en.wikipedia.org/wiki/Hyperparameter_optimization#Random_search)
3. Tree-structured Parzen Estimators (TPE) (http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf)
4. Bayesian Optimization (https://en.wikipedia.org/wiki/Hyperparameter_optimization#Bayesian_optimization)

Schedule:

7:00 - 7:30 Socializing
7:30 - 7:45 Udacity ML Nanodegree intro & fellowship awards
7:45 - 8:15 Erin Craig's talk
8:15 - 8:45 Alexandra's talk
8:45 - 9:15 Erin LeDell's talk
9:15 - 9:30 Wrap up

A big thanks to Udacity (https://www.udacity.com/) for hosting the meetup and providing food & drinks. Check out their Machine Learning Engineer Nanodegree (https://www.udacity.com/course/machine-learning-engineer-nanodegree-by-google--nd009) program.

Speaker: Erin Craig (https://www.linkedin.com/in/erincr)
Title: Hyperparameter Optimization: Grid Search and Bayesian Optimization

Abstract: When building a model, how do you select its hyperparameters? Grid search and bayesian optimization are two common methods for hyperparameter optimization; each with its own set of strengths and drawbacks. We begin this talk with a brief overview of these two methods, and then look at a case study to compare results of manual tuning, grid search and Bayesian optimization when predicting 30-day readmission from electronic health records. Code (in Python, using scikit-learn (http://scikit-learn.org/) and hyperopt (http://hyperopt.github.io/hyperopt/)) will be included.

Speaker Bio: Erin Craig is currently a machine learning fellow at Startup.ML (http://startup.ml/) and is pursuing an M.A. in data science at New College of Florida. Previously she worked at Wolfram Research, where she led teams developing math content for Wolfram|Alpha and educational software technologies. She has a B.A. in mathematics from New College of Florida.

Speaker: Alexandra Johnson (https://www.linkedin.com/in/alexandraj777)
Title: Hyperparameter Optimization 101

Abstract: Hyperparameter optimization is a common problem in machine learning. Machine learning algorithms, from logistic regression to neural nets, depend on well tuned hyperparameters to reach maximum effectiveness. Different hyperparameter optimization strategies have varied performance and cost (in time, money, and compute cycles.) So how do you choose? This talk will give a brief introduction to hyperparameter tuning and its importance, as well as the basics of how we apply statistical tests to make confident assertions about which hyperparameter optimization strategies can give you better results, faster.

Speaker Bio: Alexandra Johnson works at SigOpt (https://sigopt.com/about), which provides a lightweight API for optimal parameter tuning. She works on everything from infrastructure to product features to blog posts. Previously, she worked on growth, APIs, and recommender systems at Polyvore (acquired by Yahoo). She majored in computer science at Carnegie Mellon University with a minor in discrete mathematics and logic, and during the summers she A/B tested recommendations at internships with Facebook and Rent the Runway.

Speaker: Erin LeDell (https://twitter.com/ledell)
Title: Stacking with Random Grids in H2O

Abstract: Random grid search can be used to generate a diverse group of models from a particular algorithm class. Unlike other hyperparameter tuning methods, random grid search does not produce a sequence of models that converge towards a final "best" model. The models that are produced from a random grid search can be ensembled via stacking to produce a stronger learner than any of the constituent algorithms, even the "best" single model in the group. We will demonstrate how to perform stacking with random grids in H2O using the h2oEnsemble (https://github.com/h2oai/h2o-3/tree/master/h2o-r/ensemble) R package, as an ensemble-based alternative to hyperparameter optimization.

Speaker Bio: Erin LeDell works as a Statistician and Machine Learning Scientist at H2O.ai (http://www.h2o.ai/), an artificial intelligence company based in Mountain View, CA. At H2O.ai, she contributes to H2O (https://github.com/h2oai/h2o-3), an open source platform for scalable machine learning. She is also the founder the Bay Area WiMLDS (http://wimlds.org/chapters/about-bay-area/) meetup group, and the founder of DataScientific, Inc.