GoogleCloudPlatform / GoogleCloudPlatform/python-docs-samples

[Composer][airflow_db_cleanup] - All DAGs runs older than `max_db_entry_age_in_days` are deleted

未關閉
#13,569 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
priority: p2 samples triage me type: bug
主要語言
Jupyter Notebook
星號
8.1k
分支
6.7k
平均合併
4 天 2 小時
30 天內合併 PR
16

描述

## In which file did you encounter the issue?
`composer/workflows/airflow_db_cleanup.py`

## Did you change the file? If so, how?
```diff
dags = session.query(airflow_db_model.dag_id).distinct()
session.commit()

- list_dags = [str(list(dag)[0]) for dag in dags] + [None]
+ list_dags = [str(list(dag)[0]) for dag in dags]
for dag_id in list_dags:
query = build_query(
session=session,
```

## Describe the issue
Since [this commit](https://github.com/GoogleCloudPlatform/python-docs-samples/commit/9dc72678c09520c107e7bd9540d376c67ed6b32a#diff-167b7d6fe9453e4f3412905c9e08b577870e907f37b980115009f1b57f3c9765R375-L385) which removed a check on whether `dag_id` argument is `None`, running the airflow_db_cleanup DAG will result in deleting all DAGs runs older than `max_db_entry_age_in_days` across all DAG IDs without keeping the at least the latest run for each DAG.

As shown in the logs for the final iteration `[None]` this generates the following query:
```text
[2025-09-19T13:24:14.981+0000] {airflow_db_cleanup.py:378} INFO - Query: SELECT dag_run.state AS dag_run_state, dag_run.id AS dag_run_id, dag_run.dag_id AS dag_run_dag_id, dag_run.queued_at AS dag_run_queued_at,
dag_run.execution_date AS dag_run_execution_date, dag_run.start_date AS dag_run_start_date, dag_run.end_date AS dag_run_end_date, dag_run.run_id AS dag_run_run_id, dag_run.creating_job_id AS dag_run_creating_job_id,
dag_run.external_trigger AS dag_run_external_trigger, dag_run.run_type AS dag_run_run_type, dag_run.conf AS dag_run_conf, dag_run.data_interval_start AS dag_run_data_interval_start,
dag_run.data_interval_end AS dag_run_data_interval_end, dag_run.last_scheduling_decision AS dag_run_last_scheduling_decision, dag_run.dag_hash AS dag_run_dag_hash,
dag_run.log_template_id AS dag_run_log_template_id, dag_run.updated_at AS dag_run_updated_at, dag_run.clear_number AS dag_run_clear_number
FROM dag_run
WHERE dag_run.execution_date <= %(execution_date_1)s
```

貢獻指南

開啟貢獻指南

研究方向

從 composer/workflows/airflow_db_cleanup.py 開始,檢查 list_dags 和 dag_id 如何傳遞給 build_query。將顯示的變更與連結的 commit 進行比較,並驗證清理流程絕不會產生未限定範圍的查詢;完成的標準是移除較早的執行記錄,同時不刪除每個 DAG 保留的最新執行記錄。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
databases
Issue 類型
缺陷
難度
2/5
預估耗時
1-3 小時
活躍度
停滯
描述清晰度
描述清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。