Flip Kromer of Infochimps presentation on Ironfan: Orchestration of Chef

Flip Kromer, CTO of Infochimps and architect of the Infochimps BigData Platform will give an in-depth presentation / demo / discussion of Ironfan (previously known as cluster_chef).

Ironfan is the foundation for your Big Data stack, making provisioning and configuring your Big Data infrastructure simple. Spin up clusters when you need them, kill them when you don't, so you can spend your time, money, and engineering focus on finding insights, not getting your machines ready.

You can think of Ironfan as an orchestration layer on top of Chef. Right now its focused on deploying to AWS but will soon be able to deploy to most platforms supported by Fog and to desktop VMs like Vagrant and VMware. Work is also being done to deploy to "bare metal" as well.

Ironfan does not replace Opscode Chef, it takes Chef and powers it up to a whole new level!

Ironfan consists of the following Toolset:

ironfan-homebase (https://github.com/infochimps-labs/ironfan-homebase): centralizes the cookbooks, roles and clusters. A solid foundation for any chef user. ironfan gem (https://github.com/infochimps-labs/ironfan): core models to describe your system diagram with a clean, expressive domain-specific language knife plugins to orchestrate clusters of machines using simple commands like knife cluster launch logic to coordinate truth among chef server and cloud providers. ironfan-pantry (https://github.com/infochimps-labs/ironfan-pantry): Our collection of industrial-strength, cloud-ready recipes for Hadoop, HBase, Cassandra, Elasticsearch, Zabbix and more. silverware cookbook (https://github.com/infochimps-labs/ironfan-homebase/tree/master/cookbooks/silverware): coordinate discovery of services ("list all the machines for awesome_webapp, that I might load balance them") and aspects ("list all components that write logs, that I might logrotate them, or that I might monitor the free space on their volumes". We've been using the earlier versions of Ironfan (Cluster_chef) at Runa to deploy our own BigData stack to AWS. The Runa Stack has a lot of moving parts and I use to have a kludge of ruby scripts and yaml files to launch a cluster. Ironfan replaced all that and made it much more consistent and gave much more control.

The Runa Stack includes: Hadoop/HBase/Hive/Redis/Rails/Redis/RabbitMQ/Sensu/Graphite/Ganglia

Ironfan has always had a focus on the Hadoop Ecosystem but it is not limited to that. It ether already bundles up all the appropriate Chef Cookbooks for your stack or its very easy to mix in your own or other non infochimps cookbooks.

Ironfan allowed us to describe the cluster in a high level DSL and then "render" it to AWS. Its easy to make variations on themes and create experimental clusters. The new stuff in the latest Ironfan looks to make it even easier and more powerful.

If you have deployments that are beyond a few nodes, Ironfan should be of interest to you.

Background info:

SlideShare : Meet the Infochimps Big Data Platform
http://www.slideshare.net/infochimps

Wired.com (http://Wired.com/) : Infochimps Tame the Big-Data Elephant
http://www.wired.com/wiredenterprise/2012/02/infochimps-big-data/

[Note: SurveyMonkey has offered the meeting place at their office.They are located on the The 4th floors at at 285 Hamilton Ave, Palo Alto, CA 94301 ( http://g.co/maps/aqhaq ). The space is open (see http://360.io/JKUvxg ), has a projector and microphone equipment, and plenty of seating.
Parking is free in the garage on Ramona between Homer and Channing (see http://boomboomboom.biz/parking/ )]