apache / apache/airflow

Latest Dag lookups repeatedly deserialize cached versions

Open
#69,814 0 comments 0 reactions 0 assignees View on GitHub
area:API area:performance kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

`main` (`3.4.0`)

### What happened and how to reproduce it?

The API server's `DBDagBag.get_latest_version_of_dag` queries the latest serialized row and deserializes its payload on every lookup. This happens even when the configured LRU/TTL cache already contains the same Dag version and serialized hash.

The latest-row query is required to preserve freshness, but repeating `DagSerialization.from_dict()` for unchanged data bypasses the expensive part of the cache.

Steps to reproduce:

1. Enable the API server Dag cache with a positive `[api] dag_cache_size`.
2. Store a serialized Dag and create a process-lived `DBDagBag`.
3. Request the latest Dag by `dag_id` twice, for example through a route that calls `get_latest_version_of_dag`.
4. Observe that both requests load the latest serialized row and deserialize its payload, even though the second request resolves to the cached version and hash.

The overhead grows with the serialized payload. In a real-path benchmark covering the ORM query, Dag deserialization, and task route handler, an approximately 31 KB payload took 3.1–4.1 ms per request across SQLite, PostgreSQL, and MySQL. An approximately 305 KB payload took 28.8–30.4 ms.

### What you think should happen instead?

The API server should continue querying the latest serialized row, then reuse the cached deserialized Dag when both `dag_version_id` and `dag_hash` are unchanged. If the hash changed for the same version ID, it should discard the stale cache entry and deserialize the current row.

This preserves freshness while avoiding repeated deserialization. The same benchmark measured 0.4–1.0 ms for the approximately 31 KB payload and 1.8–3.9 ms for the approximately 305 KB payload with this behavior.

### Operating System

Reproduced in the Apache Airflow Breeze development environment on macOS.

### Deployment

Other Docker-based deployment

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start at the API server's DB dag bag method get_latest_version_of_dag and trace its call to DagSerialization.from_dict, using a route that calls this method to reproduce the repeated lookup. Keep the latest-row freshness check, reuse the cached Dag when dag_version_id and dag_hash match, and verify that changed hashes trigger deserialization instead.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.