Azure Data Engineer - Part 30: Integrate SQL and Apache Spark Pools in Synapse

Join us in this weekly series and learn how to become an Azure Data Engineer, how to integrate, transform, and consolidate data from various structured and unstructured data systems into structures that are suitable for building analytics solutions.
Responsibilities for this role include helping stakeholders understand the data through exploration, building, and maintaining secure and compliant data processing pipelines by using different tools and techniques. You will learn how to use various Azure data services and languages to store and produce cleansed and enhanced datasets for analysis.
An Azure Data Engineer also helps ensure that data pipelines and data stores are high-performing, efficient, organized, and reliable, given a specific set of business requirements and constraints. This professional deals with unanticipated issues swiftly and minimizes data loss. An Azure Data Engineer also designs, implements, monitors, and optimizes data platforms to meet the data pipeline needs.
In each week, we will cover a different module towards the full learning path, which will prepare you for the Azure Data Engineer Associate certification (https://docs.microsoft.com/en-us/learn/certifications/azure-data-engineer/) as well as for the real world.
A candidate for this certification must have a solid knowledge of data processing languages, such as SQL, Python, or Scala, and they need to understand parallel processing and data architecture patterns. Specifically, you should have the knowledge equivalent to the Azure Data Fundamentals certification.
All sessions are recorded, and the entire series can be found here: https://bit.ly/Azure-Data-Engineer-certificate.
This is part 30: Integrate SQL and Apache Spark Pools in Azure Synapse Analytics. In this module, we will learn how to integrate SQL and Apache Spark pools in Azure Synapse Analytics.
In this module, you'll learn how to:

• Describe the integration methods between SQL and Spark Pools in Azure Synapse Analytics
• Understand the use-cases for SQL and Spark Pools integration
• Authenticate in Azure Synapse Analytics
• Transfer data between SQL and Spark Pool in Azure Synapse Analytics
• Authenticate between Spark and SQL Pool in Azure Synapse Analytics
• Integrate SQL and Spark Pools in Azure Synapse Analytics
• Externalize the use of Spark Pools within Azure Synapse workspace
• Transfer data outside the Synapse workspace using SQL Authentication
• Transfer data outside the Synapse workspace using the PySpark Connector
• Transform data in Apache Spark and write back to SQL Pool in Azure Synapse Analytics

Link to the relevant module in Microsoft Learn: https://docs.microsoft.com/en-us/learn/modules/integrate-sql-apache-spark-pools-azure-synapse-analytics/
Agenda:

• 13:00-13:15 – Opening, Announcements, and More...
• 13:15-14:15 – Session
• 14:15-14:30 – Q&A and Open Discussion