apache / apache/airflow

Multi-Team: Allow creating team unique resources

Open
#72,706 1 comment 0 reactions 0 assignees View on GitHub
area:API area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

Multi team deployments use the same uniqueness constraints as regular airflow deployments, including dag_id, pool_id, variable_id and conn_id needing to be unique across the database.
This is problematic because resources for other teams are hidden, so a team will not know beforehand that a resource already exists. This creates two issues: a usability issue and a security issue. The usability issue is that two teams cannot define resources with the same id. This is a problem for teams which don't use naming conventions.

For example, both teams might want to create a dag called `load_data`, however when team 2 comes to try and define the dag they will get a parse error saying that the dag must be unique. This also creates a race condition at parse time on which team will be able to define the dag.
Another example would be two teams wanting to create a pool for spark tasks called `spark`. This is a common scenario which currently requires one team to change the name of their pool.

The security issue is that a team could use this method to guess the IDs of dags existing in other teams. This is not as important as the usability issue in my opinion but it is still worth noting.

### Use case/motivation

Update the queries and uniqueness constraints for team resources (dags/pools/variables/connections) so that they are unique within a team, and the database constraint is a pair of (id, team_name) so that two teams can define the same dags or other resources.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the models, queries, and uniqueness constraints for DAGs, pools, variables, and connections, then inspect any existing multi-team scoping tests. Done means each resource identifier is unique within its team, while different teams can use the same identifier without exposing or conflicting with one another.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.