Uber Data Platform Night

Welcome to Uber’s Data Platform Meetup!

We’re holding an Uber Data Platform Meetup to share knowledge between companies that are developing data platform solutions at scale. This evening, speakers from Google, Netflix, and Uber will present on the technologies their companies have developed to meet the challenges of making data discoverable, meaningful, and powerful, for users globally .

Agenda:

6:00 p.m. : Doors open, food and drinks

6:30 p.m. - 6:40 p.m.: Welcome and Introductions - Anthony Asta, Head of Data Platform at Uber and Jennifer Anderson, Head of Data at Uber

6:40 p.m. - 7:00 p.m.: Facilitating Discovery of Scientific Data on the Web - Natasha Noy, Staff Scientist at Google

7:00 pm, Doors close

7:00 p.m. - 7:20 p.m.: Uber Metadata: Making Data Powerful - Zoe Abrams, Engineering Manager at Uber

7:20 p.m. - 7:40 p.m.: Metacat: Making Big Data Discoverable and Meaningful at Netflix - Ajoy Majumdar, Software Architect and Zhen Li, Senior Software Engineer at Netflix

7:40 pm 8:15 p.m.: Networking

_______________________________________________________________________________________

Uber Metadata: Making Data Powerful
By: Zoe Abrams, Eng Manager at Uber

Uber Data drives the development of hundreds of global marketplaces made up of riders, drivers, and eaters, and thousands of Uber employees rely on this data to solve the challenges that come with scaling rapidly in a dynamic environment. To make this data accessible, trustworthy, and resource-efficient, we’ve built a set of Big Data tools that enable us to leverage these insights in a meaningful way. Over time, our team has bridged together a multitude of different complex data systems at Uber into one cohesive platform that builds end-to-end data lineage, integrates metadata into core workflows, and intelligently monitors data integrity issues.
_________________________________________________________________________________

Facilitating Discovery of Scientific Data on the Web
By: Natasha Noy, Staff Scientist at Google

The world generates reams of structured data every day: the data that we allow companies to collect about our everyday lives, the data that engineers produce as they analyze this collected data, the data that scientists collect to verify their next hypothesis, just to name a few examples. This data comes in many forms, formats, and sizes. Often, we want to find and, maybe, reuse the data that others have produced. Search companies today do a good job of organizing and searching unstructured data, such as HTML pages. However, organizing, understanding, searching, and reusing heterogeneous structured data is largely an open research problem. How do we collect the metadata and understand and capture the semantics of the data? How do we index and rank the datasets in search results? How do we encourage data publishers to provide semantically rich metadata? I will discuss the work that we do at Google to facilitate discovery of metadata on the Web, the challenges that we face, and our plans to contribute to the creation of a vibrant and open eco-system of data publishing.
__________________________________________________________________________

Metacat: Making Big Data Discoverable and Meaningful at Netflix
By: Ajoy Majumdar, Senior Software Engineer and Architect at Netflix and Zhen Li, Senior Software Engineer and Architect at Netflix

Most large companies have numerous data sources with different data formats and large data volumes. A centralized service is needed to provide a federated unified data and metadata access. This talk presents the design and challenges of Metacat, the metadata service in big data platform at Netflix. Metacat has the key features of data abstraction, discovery, business metadata storage, data change auditing and notifications, etc. It provides a unified REST/Thrift interface to access metadata of various data stores.