apache / apache/airflow

Optimize graph view performance by only rendering nodes in the user's view

Open
#41,620 2 comments 1 reaction 0 assignees View on GitHub
area:UI kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

The Graph view currently load all tasks in the DAG and then allows you to scroll around the graph. This is very inefficient for large DAGs because the user can only actually perceive a small fraction of the DAG at once. This inefficiency can crash the webserver (or require excessive resources) for dags with 5 figure numbers of tasks.

The prior art I would take inspiration from is video games, which do not load the whole world, but instead load only the parts that you are looking at and update that set as you move around. For Airflow, I think we should show only the tasks connected to the task you are currently looking at and the ones connected to those and so on until some limit on the total number of tasks (e.g. 50) is hit. To see the rest of the DAG, click one of the connected tasks and then Airflow should re-center the view on that task, loading in the new tasks and dropping the ones farthest away.

### Use case/motivation

Large DAGs should load intelligently in the Graph view without needing excessive resources on the webserver.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by locating Airflow's Graph view entry point and tracing how it loads and renders DAG tasks. Define the visible-task limit and recentering behavior described in the issue, then verify that large DAGs render only the connected tasks in view and update that set when the user selects another task.

Written by the indexing model from the issue text.

Assessment

Domain
frontend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.