Orchestrating ADF pipelines with managed airflow

Topic

Orchestrating ADF with Managed Apache Airflow

Abstract/Learning Objectives

Pre-requisites (Optional)

Basic knowledge of ADF pipelines

Basic Knowledge in Apache airflow

Workspace in ADF and Storage account in ADLS.

Introduction

ADF has proven to be a reliable service for orchestrating pipelines, however it does have its limitations. Fortunately, there's no need to worry. The advent of managed airflow brings a promising solution, empowering us to overcome these limitations through the power of coding.

In this challenge, you will get hands on with Airflow and you will know how to orchestrate ADF.

Description

In this challenge, you will properly set up a managed airflow instance and connect it to azure data factory.

1. Please configure the managed airflow instance with the following specifications:

Use basic auth.

Create airflow folder in your ADLS containing 2 sub-folders : dags and plugins.

2. You will have to create 2 pipelines like so: one pipeline will sum 2 random numbers

   The second pipeline will calculate the average of these 2 numbers

3. Write a DAG that will generate these two pipelines randomly – use random library in python

Success Criteria

To complete this challenge successfully, you should be able to:

Verify that the ADF has a configured airflow instance.

Verify that in the managed airflow UI you got the connection with ADF configured properly.

Demonstrate that airflow can trigger pipelines using the power of coding and see triggered pipelines in ADF monitor.

Learning Resources

How does Managed Airflow work? - Azure Data Factory | Microsoft Learn

airflow.operators.python — Airflow Documentation (apache.org)

airflow.providers.microsoft.azure — apache-airflow-providers-microsoft-azure Documentation

Advanced Challenges (Optional)

Too comfortable? Eager to do more? Try these additional challenges!

Create an alert if DAG fails (send a mail to a specific user)

Write a DAG which sends parameters to an existing pipeline in ADF.

Delivery Date

20/09/2023

Authors

No response

Other

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Start with the linked Microsoft Learn guide on Managed Airflow in Azure Data Factory, then review the Apache Airflow Python operator and Azure provider documentation. Done means a managed Airflow instance uses basic auth, its ADLS dags and plugins folders exist, two ADF pipelines calculate a sum and average, and a DAG triggers them visibly in ADF monitoring.