Stan: A Bayesian Directed Graphical Model Compiler
Details
This month we have Bob Carpenter from LingPipe and Columbia University's Department of Statistics presenting "Stan: A Bayesian Directed Graphical Model Compiler". Here'e Bob's abstract, bio, and references:
I'll present an overview of Stan, a general compiler for Bayesian directed graphical models. Users provide a model definition and specify some variables as data. Stan then draws samples of the unknown variables from the posterior distribution, which may be used for full Bayesian inference.
Stan provides an extension of the BUGS graphical modeling language. Rather than being interpreted, it is compiled to C++ for both speed and scalability.
Inference is performed using an adaptive form of Hamiltonian Monte Carlo (HMC), a Markov chain Monte Carlo (MCMC) method based on gradients of the log probability function. Gradients are calculated automatically using reverse-mode algorithmic differentiation. HMC's main advantage is being able to sample from densities with highly correlated and/or constrained variables (such as logistic regressions with correlated features or multilevel time series with covariance structure), which cause simpler methods like Gibbs sampling or random-walk Metropolis-Hastings to grind to a halt.
I'll start with an overview of the principles of Bayesian inference, including the smooth transition from supervised to semi-supervised to unsupervised models.
I'll supply examples of Stan's modeling language, describe HMC and our adaptive sampler at a high level, explain how reverse-mode algorithmic differentiation works, how we transform constrained parameters (like simplexes or covariance matrices), and how we can marginalize out discrete parameters directly in the modeling language.
I'll also discuss convergence monitoring, calculating effective sample sizes in MCMC and standard errors of MCMC estimates, as well as some automatic testing methods for probabilistic models.
I'll conclude with some future directions for research and development, including the use of second-order derivatives (Hessians) for automatic step-size tuning.
A fully functional beta version of Stan is available from Google Code under the new BSD license; version 1.0 is due early in 2012.
Joint work with: Andrew Gelman, Matt Hoffman, Daniel Lee, Wei Wang, Jiqiang Guo, Ben Goodrich, and Michael Malecki
Contact:
Bob Carpenter, carp@alias-i.com
Columbia University, Department of Statistics
LingPipe, Inc.
Bio:
Bob Carpenter works on computational linguistics, Bayesian statistics, and machine learning. He received a Ph.D. from University of Edinburgh's Centre for Cognitive Science, was an associate professor of computational linguistics at Carnegie Mellon University, and is now a research scientist in the Department of Statistics at Columbia University. Between academic stints, he was a research scientist at Lucent Bell Laboratories, a programmer and research scientist at SpeechWorks, and the lead designer and programmer for LingPipe.
References:
David Lunn, David Spiegelhalter, Andrew Thomas and Nicky Best. 2009.
The BUGS project: Evolution, critique and future directions.
Statistics in Medicine.
http://onlinelibrary.wiley.com/doi/10.1002/sim.3680/abstract
Project Home: http://www.mrc-bsu.cam.ac.uk/bugs/
Matthew D. Hoffman and Andrew Gelman. 2011.
The No-U-Turn Sampler: Adaptively Setting Path Lengths in
Hamiltonian Monte Carlo. Arxiv.
http://arxiv.org/abs/1111.4246
Radford Neal. 2011.
MCMC Using Hamiltonian Dynamics. In S. Brooks, et al.,
Handbook of Markov Chain Monte Carlo. Chapman & Hall / CRC Press.
http://www.mcmchandbook.net/HandbookChapter5.pdf
Griewank, Andreas Griewank and Andrea Walther. 2008.
Evaluating Derivatives: Principles and Techniques of Algorithmic
Differentiation. SIAM.
http://www.ec-securehost.com/SIAM/OT105.html
L. B. Rall. 1996. An introduction to automatic differentiation.
https://cs.hi.is/blasdiff/corliss-1996-ad-intro-rall.ps