Name: Nvidia GPU Half Precision FP16, INT8 +TensorRT +Spark SQL +Streaming +Tensorflow
Start: 2017-04-06T18:30:00-07:00
End: 2017-04-06T21:00:00-07:00
Location: GoDaddy

Talk 0: Meetup and Technology Updates

(Chris Fregly (https://www.linkedin.com/in/cfregly) from PipelineIO (http://pipeline.io/))

https://secure.meetupstatic.com/photos/event/1/6/1/9/600_459785657.jpeg

Part 1 of 6: Meetup Metrics

Thank You to Our New Sponsor, GoDaddy!!

I'm a big fan, obviously... 200+ Domain Names?!

https://secure.meetupstatic.com/photos/event/1/8/0/9/600_459786153.jpeg

• 6,400 Meetup Members!!

https://secure.meetupstatic.com/photos/event/1/4/2/1/600_459785153.jpeg

Part 2 of 6: PipelineIO Community Edition

• Hits 1,300+ GitHub Stars!

• 7,500+ DockerHub Downloads!!

• Global Reach in 50+ Countries on (Almost) Every Continent

https://secure.meetupstatic.com/photos/event/f/8/4/600_459783972.jpeg

Community Edition New Features

• PipelineIO CLI

• Scikit-Learn Model Serving (joins Spark, TensorFlow, PMML)

• Latest Spark and TensorFlow Versions (All Optimizations)

• Full GPU Support on both AWS and Google Cloud

AWS: https://github.com/fluxcapacitor/pipeline/wiki/AWS-GPU-Tensorflow-Docker

GCP: https://github.com/fluxcapacitor/pipeline/wiki/GCP-GPU-Tensorflow-Docker (https://github.com/fluxcapacitor/pipeline/wiki/AWS-GPU-Tensorflow-Docker)

Part 3 of 6: New PipelineIO **Premium Edition** Now Available!!

Contact: sales@pipeline.io

Features

• 24x7 Global, On-Site Support

• Premium Training Included

• Spark + Tensorflow on GPUs

• Cloud + On-Premise Hosting

• Hybrid-Cloud High Availability

• Hybrid-Cloud Cost Optimizer

• Autoscaling + Spot Pricing

• Continuous Model Training

• Model Ensemble + Optimizer

• Continuous Model Deploys

• Canary + Blue/Green Deploys

• Model A/B + Bandit Testing

• Model Server Circuit Breakers

• End-to-End Pipeline Monitoring

• Turing-Complete Metrics Queries

• Real-time Metrics Dashboards

Part 4 of 6: Upcoming PipelineIO Workshop (20% Off (https://pipelineio-distributed-tensorflow-ai-gpu-workshop.eventbrite.com?discount=ADVANCEDSPARK20))

Discount Code: ADVANCEDSPARK20 (https://pipelineio-distributed-tensorflow-ai-gpu-workshop.eventbrite.com/?discount=ADVANCEDSPARK20)

RSVP Here: https://pipelineio-distributed-tensorflow-ai-gpu-workshop.eventbrite.com?discount=ADVANCEDSPARK20 (https://pipelineio-distributed-tensorflow-ai-gpu-workshop.eventbrite.com/?discount=ADVANCEDSPARK20)

https://secure.meetupstatic.com/photos/event/1/5/1/5/600_459785397.jpeg

• All GPU-based!!!

https://secure.meetupstatic.com/photos/event/1/8/8/2/600_459786274.jpeg

• Distributed TensorFlow AI Model Training and Serving

https://secure.meetupstatic.com/photos/event/1/6/f/6/600_459785878.jpeg

• Lots of Tensorboard Visualizations

https://secure.meetupstatic.com/photos/event/1/6/b/3/600_459785811.jpeg

• Heavy Emphasis on Post-Training Graph Optimizations

https://secure.meetupstatic.com/photos/event/1/6/b/7/600_459785815.jpeg

• A Bit of Spark and Scikit-Learn ML Model Training and Serving

• A Bit of CUDA Programming -- Why Not, Right?!

Discount Code: ADVANCEDSPARK20 (https://pipelineio-distributed-tensorflow-ai-gpu-workshop.eventbrite.com/?discount=ADVANCEDSPARK20)

RSVP Here: https://pipelineio-distributed-tensorflow-ai-gpu-workshop.eventbrite.com?discount=ADVANCEDSPARK20 (https://pipelineio-distributed-tensorflow-ai-gpu-workshop.eventbrite.com/?discount=ADVANCEDSPARK20)

Part 5 of 6: O'Reilly Online Training was a Huge Success!!

High Performance Tensorflow in Production: Develop hands-on experience optimizing and deploying TensorFlow Models

https://secure.meetupstatic.com/photos/event/1/5/4/1/600_459785441.jpeg

What You Will Learn...

• The Structure of a TensorFlow Model

• To Inspect and Debug Models

• To Optimize Training with Queue Feeders

• To Optimize Training with XLA JIT Compiler

• To Optimize Inference with AOT and Graph Transforms

• The Key Components of TensorFlow Serving

• To Deploy Models with TensorFlow Serving

• To Optimize Inference by Tuning TensorFlow Serving

• AND EVERYBODY GETS A GPU!!

Thanks Google Cloud and O'Reilly!!

https://secure.meetupstatic.com/photos/event/1/0/1/0/600_459784112.jpeg

Agenda

• Setup GPU-based Environment

https://secure.meetupstatic.com/photos/event/1/0/a/a/600_459784266.jpeg

• Train and Debug TensorFlow Model

https://secure.meetupstatic.com/photos/event/1/0/a/d/600_459784269.jpeg

• Optimize TensorFlow Model Training with XLA JIT Compiler

CPUs

https://secure.meetupstatic.com/photos/event/1/0/c/8/600_459784296.jpeg

GPUs

https://secure.meetupstatic.com/photos/event/1/0/e/4/600_459784324.jpeg

• Optimize Trained Model with XLA AOT and Graph Transforms

fold_batch_norms

fold_constants

merge_duplicate_nodes

strip_unused_nodes

remove_device

round_weights

quantize_weights (diagram below)

https://secure.meetupstatic.com/photos/event/1/5/d/6/600_459785590.jpeg

• Deploy Model to TensorFlow Serving Runtime

TensorFlow Serving Architecture

https://secure.meetupstatic.com/photos/event/1/2/4/4/600_459784676.jpeg

Tensorflow Serving Inputs and Outputs

https://secure.meetupstatic.com/photos/event/1/2/7/a/600_459784730.jpeg

• Optimize TensorFlow Serving Runtime

Enable GPU Support for TensorFlow Serving
(Requires building from source with mystery parameters)

https://secure.meetupstatic.com/photos/event/1/2/5/9/600_459784697.jpeg

Request Batching

https://secure.meetupstatic.com/photos/event/1/2/8/c/600_459784748.jpeghttps://secure.meetupstatic.com/photos/event/1/2/a/9/600_459784777.jpeg

NetflixOSS-based ML/AI Model Servers with Circuit Breakers
(Highly-scalable and available. Lots of metrics.)

https://secure.meetupstatic.com/photos/event/1/2/6/1/600_459784705.jpeg

Asynchronous, Python-based Http Server
(Like Flask, but More Complicated and, therefore, Cooler)

https://secure.meetupstatic.com/photos/event/1/2/5/0/600_459784688.jpeg

Part 6 of 6: Google's New Tensor Processing Units (TPUs)

• https://techcrunch.com/2017/04/05/google-says-its-custom-machine-learning-chips-are-often-15-30x-faster-than-gpus-and-cpus/

• https://cloudplatform.googleblog.com/2017/04/quantifying-the-performance-of-the-TPU-our-first-machine-learning-chip.html

• https://drive.google.com/file/d/0Bx4hafXDDq2EMzRNcy1vSUxtcEk/view

https://tctechcrunch2011.files.wordpress.com/2017/04/2017-04-05_1014.png

https://tctechcrunch2011.files.wordpress.com/2017/04/2017-04-05_1011.png

Talk 1: Deep Dive into Spark SQL's JsonPath Evaluator, Jackson Streaming, Scala Parser Combinators (Nathan Howell (https://www.linkedin.com/in/nathanhowell/), Architect @ GoDaddy's Global Platform Team, Spark Contributor, Tensorflow Contributor)

We’ll explore how JsonPath expressions are parsed and evaluated one token at a time with Spark’s smallest interpreter. Learn about parser combinators in Scala, Jackson Streaming, and how compatibility with Hive's adhoc implementation was retained.

Speaker Bio

Nathan Howell (https://www.linkedin.com/in/nathanhowell/) is an architect in GoDaddy’s Global Platform team. He’s been contributing to Spark since 2014 and is an avid Tensorflow-user. Prior to joining GoDaddy he spent 15 years toiling away on large distributed machine learning systems at Microsoft, eBay and a few failed startups.

Talk 2: Reduced Precision (FP16, INT8) Inference on Convolutional Neural Networks with TensorRT and NVIDIA Pascal (Chris Gottbrath (https://www.linkedin.com/in/chrisgottbrath/), Nvidia (http://www.nvidia.com))

NVIDIA’s Pascal GPUs provide developers a platform for both training and deploying neural networks. In deployment GPUs allow lower latencies or servicing large inference workloads with a smaller set of accelerated nodes. One advanced technique to optimize throughput is to leverage the Pascal GPU family’s reduced precision instructions. I’ll show how you can start with a network trained in FP32 and deploy that same network with 16 bit or even 8 bit weights and activations using TensorRT. I’ll talk in some detail about the mechanics of converting a neural network and what kinds of performance and accuracy we are seeing on image net style networks.

I’ll end with a quick overview of how developers can deploy these DL networks as micro services using the GPU REST Engine.

References

• https://devblogs.nvidia.com/parallelforall/deploying-deep-learning-nvidia-tensorrt/

Talk 3: High-Performance Tensorflow Serving across Hybrid Cloud and On-Premise + Request Batching + Model Post-Processing Optimizations (Chris Fregly (https://www.linkedin.com/in/cfregly) from PipelineIO (http://pipeline.io/))

In this completely demo-based talk, Chris will demonstrate various techniques to post-process and optimize trained Tensorflow AI models to reduce deployment size and increase prediction performance.

First, we'll use various techniques such as 8-bit quantization, weight-rounding, and batch-normalization folding, we will simplify the path of forward propagation and prediction.

Next, we'll loadtest and compare our optimized and unoptimized models - in addition to enabling and disabling request batching.

Last, we'll dive deep into Google's Tensorflow Graph Transform Tool (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/graph_transforms/README.md) to build custom model optimization functions.

Speaker Bio

Chris Fregly (https://www.linkedin.com/in/cfregly) is a Research Scientist at PipelineIO (http://pipeline.io/) - a Machine Learning and Artificial Intelligence Startup in San Francisco.

Chris is an Apache Spark Contributor, Netflix Open Source Committer, Founder of the Global Advanced Spark and TensorFlow Meetup, and Author of the O'Reilly Training and Video Series, "High Performance TensorFlow in Production"

Previously, Chris was a Distributed Systems Engineer at Netflix, a Data Solutions Engineer at Databricks, and a Founding Member of the IBM Spark Technology Center in San Francisco.

Chris Fregly

AI Performance Engineering Meetup (San Francisco, Global)

Technology

Predictive Analytics

High Scalability Computing

Artificial Intelligence Programming

Artificial Intelligence Applications

Apache Spark

Data Science

Machine Learning

TensorFlow

Big Data

Deep Learning

Neural Networks

Artificial Intelligence

Natural Language Processing

Kubernetes

PyTorch

Nvidia GPU Half Precision FP16, INT8 +TensorRT +Spark SQL +Streaming +Tensorflow

GoDaddy

Share

AI Performance Engineering Meetup (San Francisco, Global)

Nvidia GPU Half Precision FP16, INT8 +TensorRT +Spark SQL +Streaming +Tensorflow

AI Performance Engineering Meetup (San Francisco, Global)

Details

Members are also interested in