apache / apache/airflow

Add support to `--build-constraint(s)` flag for our constraint preparation

Open
#54,394 2 comments 0 reactions 0 assignees View on GitHub
area:dev-env kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

Currently in airflow CI process we prepare constraints automatically and both our CI process and user installation are designed around using it to "stabilize" both CI (PR stability) and Reproducible installation of airflow.

This prevents Airflow from being susceptible to breaking installation and tests when new version of a dependency (including transitive dependencies) is released. We can detect such breakages and remedy and report them thanks to our canary builds, without impacting stability of regular PRs. Also our users can (almost - see below) reliably install airflow reproducibly.

However (as evidenced by https://github.com/pypa/setuptools-scm/issues/1194 and https://github.com/pypa/setuptools-scm/issues/1191 - when installing packages, the build isolation feature causes that installing 3rd-party packages from sources might be broken simply by a fact of releasing their build dependencies.

Uv already has `--build-constraint` flag and there is work on-going on adding similar flag to `pip` - https://github.com/pypa/pip/pull/13534, and we should be able to come out with automation that gathers information about current version of build tools used for all our ".sdist" dependencies and freezes it to a separate "build-constraints" files alongside regular constraints.

That should make us resilient to future cases like that (while still allowing us to quickly detect and report such cases with our canary builds.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Airflow’s existing CI constraint-preparation process and how its sdist dependencies are collected. Compare uv’s --build-constraint behavior with the referenced pip work, then determine how build-tool versions should be gathered into separate build-constraints files. Done means the automation produces those files while preserving regular constraints and allowing canary builds to detect failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, ci-cd
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.