Docker: new approaches to software development

Agenda:

6:00 - Networking & Pizza.

6:45 - Welcome. Members to share topics of interests for future meetings.

7:00 - Talk #1.

7:30 - Q&A break.

7:45 - Talk #2.

8:15 - Q&A break & wrap-up.

Talk #1: Deploying Hadoop Using Docker Containers: What Works and What Doesn’t

In this presentation, Nasser Manesh will share his experiences and lessons learned in deploying multi-tenant Hadoop clusters on top of Docker containers.

● Containers vs Virtual Machines
Why it makes sense to use containers, rather than virtual machines, to provide resource
limits and isolation for Hadoop components
● Key issues with containers
What can go wrong if one uses containers with Hadoop?
● Hadoop in Docker, or Docker in Hadoop?
Understanding the two main models to use Docker with Hadoop.
● Resource allocation and configuration management:
How to configure containers for a datanode vs a nodemanager vs a namenode, finding the optimal number of containers per machine, and sizing containers.
● Monitoring, Metrics, and Troubleshooting:
How to make Hadoop know about the health of the Docker containers in order to distribute jobs properly, and how to collect for proper reporting on resource utilization.
● Disk and network access:
How to make sure that Docker can access only certain parts of the disk allocated to HDFS, and how to make containers leverage the high-bandwidth, high-throughput network infrastructure purpose-built for Hadoop.

Presenter: Nasser Manesh, Senior Engineer, Infrastructure/Operations
Altiscale, Inc.

Nasser Manesh has 25+ years of experience in Unix, infrastructure, distributed systems, and backend operations in DevOps, team lead, and CTO roles. He has founded startups in consumer Internet, mobile, photography and art areas. Nasser is currently focused on Big Data infrastructure, Hadoop core (HDFS/YARN), Chef, Linux cgroups, and Docker at scale, and is a senior operations engineer at Altiscale which provides Hadoop as a Service in the cloud.

Talk #2: Dockerizing Your Organization

In his presentation, Dan Lorenc will cover why, when and how to introduce Docker into your development practices.

Why

• How Docker can save you time setting up local development environments

• How Docker can make your local environments closer to production

When

• When is the best time to rewrite your development tooling to use Docker?

How

• How to teach your team to use Docker

• What Docker tools to use and when (machine vs. boot2docker vs. compose vs. fig)

• How to stay up to date with docker best practices

• Common problems your team will encounter

Presenter: Dan Lorenc, Software Engineer at Google

Dan Lorenc is a software engineer experienced in building developer tools in Python, Go and .Net. He has worked at companies ranging from small startups to Microsoft and Google. Dan is currently focused on bringing the power and flexibility of Docker to Google App Engine, a PaaS product. Dan also wrote the Google Compute Engine driver for the new Docker Machine tool.