Concurrent POST /api/v2/backfills causes HTTP 500 + partial data with SQLite metadata DB
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.2.2
### What happened and how to reproduce it?
Apologies for AI slop but the issue is clearly reproducible.
`<🤖> `
---
Sending multiple concurrent `POST /api/v2/backfills` requests for the same DAG results in HTTP 500 with empty response body, but those failed requests partially commit `backfill_dag_run` rows to SQLite. The partially-created backfill then blocks future creation with "already running backfill".
Root cause visible in API server logs:
sqlite3.OperationalError: database is locked
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
`_create_runs_non_partitioned` inserts rows in a loop using `begin_nested()` savepoints. When SQLite's busy timeout fires mid-loop the exception propagates as HTTP 500, but rows flushed by earlier savepoints survive — leaving a partially-initialised backfill.
My mise.toml
```tomls
[tools]
python = "3.12"
[env]
_.path = [".venv/bin"]
AIRFLOW_HOME = "{{config_root}}"
AIRFLOW__API__PORT = "48001"
AIRFLOW__CORE__EXECUTION_API_SERVER_URL = "http://localhost:48001/execution/"
AIRFLOW__CORE__LOAD_EXAMPLES = "false"
AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL = "1"
AIRFLOW__LOGGING__WORKER_LOG_SERVER_HOST = "127.0.0.1"
AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP = "15"
AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG = "15"
AIRFLOW__CORE__PARALLELISM = "50"
[tasks.setup]
description = "Install Airflow into venv"
run = """
pip install "apache-airflow==3.2.2" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.2.2/constraints-no-providers-3.12.txt"
"""
```
**To reproduce:**
1. Create a minimal DAG (`test_backfill_bug.py`):
from airflow.sdk import dag, task
from datetime import datetime
@dag(
dag_id="test_backfill_bug",
schedule="@daily",
start_date=datetime(2020, 1, 1),
end_date=datetime(2022, 12, 31),
catchup=False,
)
def test_backfill_bug():
@task
def noop():
pass
noop()
test_backfill_bug()
2. Fire 5 concurrent backfill creation requests:
TOKEN=$(curl -s -X POST "http://localhost:8080/auth/token" \
-H "Content-Type: application/json" \
-d '{"username": "admin", "password": "admin"}' \
| python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
for i in 1 2 3 4 5; do
curl -s -o /dev/null -w "[%{http_code}] " \
-X POST "http://localhost:8080/api/v2/backfills" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"dag_id":"test_backfill_bug","from_date":"2020-01-01T00:00:00Z","to_date":"2022-12-31T00:00:00Z","max_active_runs":1,"reprocess_behavior":"none"}' &
done
wait
3. Check SQLite for partial data:
python3 -c "
import sqlite3, os
conn = sqlite3.connect(os.path.expanduser('~/airflow/airflow.db'))
cur = conn.cursor()
cur.execute('SELECT id, created_at FROM backfill WHERE dag_id=\"test_backfill_bug\" ORDER BY id')
for b_id, created_at in cur.fetchall():
cur.execute('SELECT COUNT(*) FROM backfill_dag_run WHERE backfill_id=?', (b_id,))
print(f'backfill {b_id}: {cur.fetchone()[0]}/1096 runs created_at={created_at}')
conn.close()
"
Observed:
[201] [500] [500] [409] [409]
backfill 1: 1096/1096 runs created_at=...
backfill 2: 2/1096 runs created_at=... <- HTTP 500 but partial data committed
### What you think should happen instead?
Either full success (201) or a clean failure with zero rows written. A request that returns HTTP 500 should never leave partial data in the database. Currently the partially-created backfill cannot be retried (409 "already running") and must be manually cleaned up from SQLite.
### Operating System
macOS
### Deployment
Virtualenv installation
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
Distinct from #61375 (premature completion, fixed in 3.2.2). This is a data integrity issue in the creation path — `_create_runs_non_partitioned` in `airflow/models/backfill.py` has no protection against partial commit on SQLite lock timeout.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start at the POST /api/v2/backfills entry point and inspect _create_runs_non_partitioned in airflow/models/backfill.py, focusing on the nested savepoints and SQLite lock failure described in the logs. Reproduce the concurrent requests with the supplied DAG and curl commands, then verify that a 500 response leaves zero backfill_dag_run rows and that successful requests remain complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlalchemy, sqlite
- Domain
- api, backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100