Past Meetup

Using Apache Airflow to Manage Big Data & ML Operations in the Cloud

This Meetup is past

12 people went

Needs a location


To confirm your place, please RSVP in this separate link:

Join us for delicious food & drinks while we discuss solutions to today's most common Data Engineering challenges faced today as companies now need to apply machine learning techniques on their data in order to remain relevant. Among the new challenges faced by data engineers is the need to build and fill Data Lakes as well as reliably delivering complete large-volume data sets so that data scientists can train more accurate models.

Aside from dealing with larger data volumes, these pipelines need to be flexible in order to accommodate the variety of data and the high processing velocity required by the new ML applications. We will end with a real world Movie Recommendation Engine example pipeline, showing how Qubole addresses these challenges by providing an auto-scaling cloud-native platform to build and run these data pipelines.

At this event, we will cover:
- Some of the typical challenges faced by data engineers when building pipelines for machine learning.
- Typical uses of the various Qubole engines to address these challenges.
- Real-world customer examples