Optimize graph view performance by only rendering nodes in the user's view
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
The Graph view currently load all tasks in the DAG and then allows you to scroll around the graph. This is very inefficient for large DAGs because the user can only actually perceive a small fraction of the DAG at once. This inefficiency can crash the webserver (or require excessive resources) for dags with 5 figure numbers of tasks.
The prior art I would take inspiration from is video games, which do not load the whole world, but instead load only the parts that you are looking at and update that set as you move around. For Airflow, I think we should show only the tasks connected to the task you are currently looking at and the ones connected to those and so on until some limit on the total number of tasks (e.g. 50) is hit. To see the rest of the DAG, click one of the connected tasks and then Airflow should re-center the view on that task, loading in the new tasks and dropping the ones farthest away.
### Use case/motivation
Large DAGs should load intelligently in the Graph view without needing excessive resources on the webserver.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by locating Airflow's Graph view entry point and tracing how it loads and renders DAG tasks. Define the visible-task limit and recentering behavior described in the issue, then verify that large DAGs render only the connected tasks in view and update that set when the user selects another task.
Written by the indexing model from the issue text.
Assessment
- Domain
- frontend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100