Latest Dag lookups repeatedly deserialize cached versions
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
`main` (`3.4.0`)
### What happened and how to reproduce it?
The API server's `DBDagBag.get_latest_version_of_dag` queries the latest serialized row and deserializes its payload on every lookup. This happens even when the configured LRU/TTL cache already contains the same Dag version and serialized hash.
The latest-row query is required to preserve freshness, but repeating `DagSerialization.from_dict()` for unchanged data bypasses the expensive part of the cache.
Steps to reproduce:
1. Enable the API server Dag cache with a positive `[api] dag_cache_size`.
2. Store a serialized Dag and create a process-lived `DBDagBag`.
3. Request the latest Dag by `dag_id` twice, for example through a route that calls `get_latest_version_of_dag`.
4. Observe that both requests load the latest serialized row and deserialize its payload, even though the second request resolves to the cached version and hash.
The overhead grows with the serialized payload. In a real-path benchmark covering the ORM query, Dag deserialization, and task route handler, an approximately 31 KB payload took 3.1–4.1 ms per request across SQLite, PostgreSQL, and MySQL. An approximately 305 KB payload took 28.8–30.4 ms.
### What you think should happen instead?
The API server should continue querying the latest serialized row, then reuse the cached deserialized Dag when both `dag_version_id` and `dag_hash` are unchanged. If the hash changed for the same version ID, it should discard the stale cache entry and deserialize the current row.
This preserves freshness while avoiding repeated deserialization. The same benchmark measured 0.4–1.0 ms for the approximately 31 KB payload and 1.8–3.9 ms for the approximately 305 KB payload with this behavior.
### Operating System
Reproduced in the Apache Airflow Breeze development environment on macOS.
### Deployment
Other Docker-based deployment
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start at the API server's DB dag bag method get_latest_version_of_dag and trace its call to DagSerialization.from_dict, using a route that calls this method to reproduce the repeated lookup. Keep the latest-row freshness check, reuse the cached Dag when dag_version_id and dag_hash match, and verify that changed hashes trigger deserialization instead.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100