Multi-Team: Allow creating team unique resources
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
Multi team deployments use the same uniqueness constraints as regular airflow deployments, including dag_id, pool_id, variable_id and conn_id needing to be unique across the database.
This is problematic because resources for other teams are hidden, so a team will not know beforehand that a resource already exists. This creates two issues: a usability issue and a security issue. The usability issue is that two teams cannot define resources with the same id. This is a problem for teams which don't use naming conventions.
For example, both teams might want to create a dag called `load_data`, however when team 2 comes to try and define the dag they will get a parse error saying that the dag must be unique. This also creates a race condition at parse time on which team will be able to define the dag.
Another example would be two teams wanting to create a pool for spark tasks called `spark`. This is a common scenario which currently requires one team to change the name of their pool.
The security issue is that a team could use this method to guess the IDs of dags existing in other teams. This is not as important as the usability issue in my opinion but it is still worth noting.
### Use case/motivation
Update the queries and uniqueness constraints for team resources (dags/pools/variables/connections) so that they are unique within a team, and the database constraint is a pair of (id, team_name) so that two teams can define the same dags or other resources.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by tracing the models, queries, and uniqueness constraints for DAGs, pools, variables, and connections, then inspect any existing multi-team scoping tests. Done means each resource identifier is unique within its team, while different teams can use the same identifier without exposing or conflicting with one another.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100