apache / apache/airflow

Split out providers into "standalone" python packages

Open
#33,909 6 comments 2 reactions 0 assignees View on GitHub
kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Body

As part of making it possible to move out of providers from Airflow core repository (we have not decided yet on it, we just want to make it possible) we should turn Airflow Providers into "real" packages.

Currently those packages are build "dynamically" -> https://github.com/apache/airflow/blob/main/dev/provider_packages/prepare_provider_packages.py is used as part of `breeze release-management prepare-provider-packages` to extract parts of the "airflow/providers/" sources dynamically, generate setup.* and pyproject.toml files dynamically and build the providers from those dynamically generated temporary folders.

This has some disadvantages - for example it does not make reproducible builds possible, and it requires complex `breeze` command, CI image and the python script in the image to build the packages (CI image is used to make sure all dependencies are installed and to provide isolation and cleanup between builds, also it allows to isolate (security) the host from container building the packages whenbuilding the providers in case of builds from contributor forks).

But this got us through last 3 years of releasing airflow and providers separately :).

With recent changes (#32604 upcoming #32048, the upcoming #33907 and a number of other changes already implemented in the past - we are quite close to make it possible to split out providers to "standalone" packages - where each provider is a separate "compliant with standard" package and has a complete independent directory where you can build the package without moving the sources around, using standard python tooling. This would require everything that relates to the providers to move to those directories (docs, tests are notably shared between airflow and providers in "docs" and "tests" and they should be moved around).

We should also eventually add automation of checking if there is anything left in core that refers to providers #11435.

The early draft POC attempts to add scripts to automate such migration and result of it can be seen here:

* https://github.com/apache/airflow/pull/28291
* https://github.com/apache/airflow/pull/28292

However, the challenge to solve for this one is to make it easy for contributors to contribute to airflow and providers together. We want to make it possible - simlarly as today to have an easy environment where you can edit both airflow and provider code and run tests, run airflow, run integration tests without extra hassle of installing and reinstalling the packages in editabkle mode.

Ideal workflow of the developer is where they can:

* run `breeze` and be able to edit any sources locally in the host - and airflow in breeze should automatically pick the changes in both airlfow and providers when interpreter is started
* install a local venv (in editable mode possibly) for airlfow and selected (all?) providers and the locally installed airflow should also pick changes for both airflow and providers when interpreter is started

The way how to do it and choice of (ideally) standard python tooling to make such move is not yet determined and is open for discussion, POCs and proposals.

Note: Some of the modern tools from PyPA world - [Hatch](https://github.com/pypa/hatch), [flit](https://github.com/pypa/flit) are recently evolving and adding more features, which might likely make it possible to combine multiple packages from monorepo into a single development installation, and we would likely want to use one of the standard tools for that rather than develop our own. We might consider contributing to some of those tools to make them more suitable for us. Possibly we could combine several tools (and for example use flit for providers, and hatch for airflow to combine repos as hatch seems to be better suited and has a roadmap for monorepo/multi-project setup, while flit is slick and very focused - or so it seems).

### Committer

- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by reading dev/provider_packages/prepare_provider_packages.py and the breeze release-management prepare-provider-packages workflow, then review the proof-of-concept pull requests #28291 and #28292. The work is complete when a documented, standard packaging approach and proof of contributor workflows support independent provider builds while allowing Airflow and provider changes to be edited and tested together.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.