apache / apache/airflow

Drop airflow-core's own dev-group dependency on 6 providers

Open
#71,641 1 comment 0 reactions 0 assignees View on GitHub
area:core
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

## What

`airflow-core/pyproject.toml`'s own `dev` dependency-group hardcodes 6 providers so that `uv sync --project airflow-core` (a scoped, core-only sync) can still run the full `airflow-core` test suite:

```toml
[dependency-groups]
dev = [
"apache-airflow-core[all]",
"apache-airflow-ctl",
"apache-airflow-devel-common",
"apache-airflow-task-sdk",
# TODO(potiuk): eventually we do not want any providers nor apache-airflow extras to be needed for
# airflow-core tests
"apache-airflow[pandas,polars]",
"apache-airflow-providers-amazon",
"apache-airflow-providers-celery",
"apache-airflow-providers-cncf-kubernetes",
"apache-airflow-providers-fab>=2.2.0",
"apache-airflow-providers-git",
"apache-airflow-providers-ftp",
]
```

There's an inline TODO already acknowledging this should go away "eventually," but no tracking issue exists for it.

## Why this matters

This is the remaining piece of the "standalone core testing" goal from #60770. That issue reported that `uv sync --project airflow-core` succeeds but the test suite doesn't actually run clean in that scoped environment. #71637 fixed the concrete, reproducible failures (a missing shared-package import and 11 tests that hard-coded full-provider-set assumptions), so the test suite now passes in this scoped sync — but only because these 6 providers are still installed alongside airflow-core's own dependencies. The environment isn't actually provider-free yet.

## Scope of the follow-up

1. Identify which `airflow-core` tests genuinely need `amazon`, `celery`, `cncf-kubernetes`, `fab`, `git`, and `ftp` (e.g. via `apache.hive`-style plugin tests, FAB-based auth-manager tests, connection form widgets that only render via `flask_appbuilder`, etc.).
2. For each: either move the test into the relevant provider's own test suite (if it's really testing provider behavior that leaked into `airflow-core`), or make it skip/adapt gracefully when that specific provider isn't installed (matching the pattern used for the tests fixed in #71637).
3. Once nothing in `airflow-core/tests` needs a provider unconditionally, drop the 6 provider entries (and `apache-airflow[pandas,polars]`) from `airflow-core/pyproject.toml`'s `dev` group and remove the TODO comment.
4. Update `contributing-docs/07_local_virtualenv.rst`'s existing note about standalone core development not being fully supported, once resolved.

This is a larger, multi-PR migration (touching many test files across `fab`/`amazon`/`celery`/`cncf.kubernetes`/`git`/`ftp`-adjacent behavior) rather than a single mechanical change, which is why it's split out from #60770 instead of being done in the same PR.

related: #60770

Contributor guide

Open the contributing guide

Research direction

Start with airflow-core/pyproject.toml, airflow-core/tests, and the standalone-core testing changes from #71637. Identify tests that unconditionally need the six listed providers or apache-airflow extras, then move or adapt them as appropriate and verify the scoped core-only test suite passes without them. Finish by removing those dependencies and the TODO, and update contributing-docs/07_local_virtualenv.rst.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, developer-experience, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.