PyData Bristol - 28th Meetup - 16th of May!

Date set!

Join us once again for the next PyData Bristol Meetup!
We’re meeting in a lovely venue Amdaris!! Big thank you for hosting our event this time.
Massive thanks to our other sponsor - Adlib for the scrumptious pizza, and thirst-quenching refreshments.
Here is an intro from out new hosts!

Hi, We are Amdaris, an Insight company, and at our core we specialise in extending teams with highly skilled software experts. With bespoke Software Development, Product Design, Strategy and Consultation, Managed Services and Data Solutions, we seamlessly integrate into your business and culture, bringing passion, care, and technical proficiency directly to you.

Agenda for the evening:
🚪 6:00 pm - Doors open
🕡 6:30 pm - Talks commence (sharp!)
📚 Deep Dive 50 Minute Talk:

• Better ETL with Managed Airflow in ADF by Niall Langley

📢 Community announcements
🤝 Relaxed networking over beers and soft drinks
Interested in sharing your knowledge or experience at this or a future event? Fill out this form to submit your talk proposal: PyData Bristol Talk Proposal
We look forward to seeing you there for another fantastic evening of Python, Data Science, and camaraderie!
‌
‌

# Talks

Better ETL with Managed Airflow in ADF by Niall Langley
Building complex data workflows using Azure Data Factory can get a little clunky - as you orchestration needs get more complex you hit limitations like not being able to nest loops or conditionals, running simple Python, bash or PowerShell scripts is difficult, and costs can grow quickly as you are charged per task execution. Recently another option become available, Managed Airflow in ADF.
Apace Airflow is a code-centric open-source platform for developing, scheduling and monitoring batch-based data workflows, built using the python language Data Engineers know and love. But until Managed Airflow, getting it working in Azure was a complex task for customers more used to PaaS services such as ADF, Databricks and Fabric. It is also an important ETL orchestrator on AWS and GCP, so cross cloud compatibility becomes simpler to achieve.
In this session we’ll look at what Airflow is, how it’s different from ADF, and what advantages Managed Airflow in ADF gives us. We talk about the idea of a DAG for building the workflow, and then work through some demos to show just how easy it is to use Python to write an Airflow DAG’s and import them into the Managed Airflow Environment as pipelines. We then dive into the excellent monitoring UI and find out just how easy is it to trigger a pipeline, view it to see the dependencies between tasks, and monitor runs.
By the end of the session attendees will have a good understanding of what Airflow is, when to use it, and how it fits into the Azure Data Platform.

#

‌
🕖 LOGISTICS
Talks kick off at 18:30 sharp; then networking in Left Handed Giant from 20:40.
If you realise you can't make it, please un-RSVP in good time to free up your place for your fellow community members.
Follow @‌pydatabristol (PyData Bristol (@PyDataBristol) / X) for updates on this and future events, as well as news from the global PyData community.
📜 CODE OF CONDUCT
The PyData Code of Conduct governs this meetup (Code of Conduct
). To discuss any issues or concerns relating to the code of conduct or behaviour of anyone at the PyData meetup, please contact the PyData Bristol organisers, or you can submit a report of any potential Code of Conduct violation directly to NumFOCUS (NumFOCUS Code of Conduct Report Form).

## Speakers

Niall Langley