Past Meetup

State-of-the-Art Deep Learning on Apache Spark and Introduction to Envoy proxy

This Meetup is past

218 people went

Location image of event venue


We are very excited to have invited Dr. Xiangrui Meng, Long time Apache Spark PMC to talk about the State-of-the-Art Deep Learning on Spark. We also invited Mark McBride to talk about Envoy proxy as an intermediary for traffic control in Microservice, Kubernate services.

6 pm -- 6:30 pm, Networking + Light dinner
7 pm -- door will be closed at 7pm

6:30 pm -- 6:40pm Introduction
6:40 pm -- 7:30 pm Talk 1 (Xiangrui Meng)
7: 35 pm -- 8:15 pm Talk 2 (Mark McBride)
8:15 pm -- 8:30 pm closing
8:45 pm office closed.

Talk 1:
Project Hydrogen: State-of-the-Art Deep Learning on Apache Spark (Databricks)

Abstract: Big data and AI are joined at the hip: the best AI applications require massive amounts of constantly updated training data to build state-of-the-art models AI has always been on of the most exciting applications of big data and Apache Spark. Increasingly Spark users want to integrate Spark with distributed deep learning and machine learning frameworks built for state-of-the-art training. On the other side, increasingly DL/AI users want to handle large and complex data scenarios needed for their production pipelines.

This talk introduces a new project that substantially improves the performance and fault-recovery of distributed deep learning and machine learning frameworks on Spark. We will introduce the major directions and provide progress updates, including 1) barrier execution mode for distributed DL training, 2) fast data exchange between Spark and DL frameworks, and 3) accelerator-awareness scheduling.

Bio: Xiangrui Meng is an Apache Spark PMC member and a software engineer at Databricks. His main interests center around developing and implementing scalable algorithms for scientific applications. He has been actively involved in the development and maintenance of Spark MLlib since he joined Databricks. Before Databricks, he worked as an applied research engineer at LinkedIn, where he was the main developer of an offline machine learning framework in Hadoop MapReduce. His Ph.D. work at Stanford is on randomized algorithms for large-scale linear regression problems.

Talk 2: Introduction to Envoy proxy (Turbine Labs)

As microservices become more widespread, managing the communication between them becomes more challenging. Using the Envoy proxy as an intermediary gives you some great traffic control capabilities. We'll walk through detailed examples of how and why to use them.

As your kubernetes service footprint grows, adding traffic control capabilities beyond kube-proxy becomes critical. Envoy provides fine grained routing control, load shedding, and metrics that help you scale your environment smoothly. We'll walk through several traffic control strategies using Envoy.

Mark McBride is founder and CEO of Turbine Labs, makers of Houston and Rotor, a modern traffic management plane. Prior to Turbine Labs he ran server side engineering at Nest. Before that he worked at Twitter, working on migrating their rails code base to JVM-based equivalents.