Deploying BERT in Production using the TensorRT Inference Server on Google Cloud
Details
Description: This talk will dive into the features and functionality of the NVIDIA TensorRT Inference Server, a microservice built for production inference to maximize the utilization of GPUs by using features like concurrent model execution and dynamic batching. Other features include multiple model framework support, custom backend support for customized implementations, and model ensembling for model stitching and pipelining. This talk will cover the details of these features along with other performance enhancements like the use of shared memory to reduce network overhead, the model control API for model loading and unloading, and the streaming API to support stateful model use cases like ASR.
This talk will also show how you can deploy BERT in an instance of TensorRT Inference Server in on GCP. Models are stored in Google Cloud Storage and fed into the inference server. A GKE cluster will also be used for the inference server deployment where the client is set up in a jupyter notebook to send requests to the server for inference.
Speaker Bio: David Goodwin is a principal software engineer in the Machine Learning group at NVIDIA where he is currently working on tools and usability for deep learning inference.
Speaker Bio: Davide Onofrio is a Senior Deep Learning SW Technical
Marketing Engineer at NVIDIA. He is involved in the production and presentation of deep learning technical developer focused-content at NVIDIA. He worked many years as computer vision and machine learning engineer in Biometrics, VR and Automotive industry. His educational background includes a PhD in signal processing at the Politecnico di Milano.
Agenda
5:45 - 6:30 PM Networking/Social/Food
6:30 - 6:35 PM Introduction
6:35 - 7:30 PM Talk and Q&A
7:30 - 8:00 PM Networking/Social/Closing
