DAG disappears in airflow 3.1.3
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
3.1.3
### If "Other Airflow 2/3 version" selected, which one?
3.1.3
### What happened?
What happened?
After upgrade to airflow 3.1.3, system started experiencing random DAG disappearance.
The config for dag processor has this setup:
```
- name: AIRFLOW__DAG_PROCESSOR__BUNDLE_REFRESH_CHECK_INTERVAL
value: '120'
- name: AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL
value: '120'
- name: AIRFLOW__CORE__STORE_SERIALIZED_DAGS
value: 'True'
- name: AIRFLOW__CORE__MIN_SERIALIZED_DAG_UPDATE_INTERVAL
value: '60'
- name: AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL
value: '120'
- name: AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD
value: '86400'
```
Files found by aiflow dag -folder also reduce over the period of time.
Here is my dag processor values.yaml file section-
```
dagProcessor:
enabled: true
replicas: 1
revisionHistoryLimit: ~
command: ~
args: ["bash", "-c", "exec airflow dag-processor"]
strategy:
rollingUpdate:
maxSurge: "100%"
maxUnavailable: "50%"
livenessProbe:
initialDelaySeconds: 120
timeoutSeconds: 60
failureThreshold: 10
periodSeconds: 60
command: ~
serviceAccount:
automountServiceAccountToken: true
create: false
name: "airflow"
annotations: {}
securityContext: {}
securityContexts:
pod: {}
container: {}
containerLifecycleHooks: {}
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 500m
memory: 500Mi
terminationGracePeriodSeconds: 60
safeToEvict: true
extraContainers: []
extraInitContainers: []
extraVolumes: []
extraVolumeMounts: []
# Select certain nodes for airflow dag processor pods.
nodeSelector: {}
affinity: {}
tolerations: []
topologySpreadConstraints: []
priorityClassName: ~
annotations: {}
podAnnotations: {}
logGroomerSidecar:
enabled: true
command: ~
args: ["bash", "/clean-logs"]
retentionDays: 15
frequencyMinutes: 15
resources: {}
securityContexts:
container: {}
env: []
waitForMigrations:
enabled: true
env: []
securityContexts:
container: {}
env:
- name: AIRFLOW__DAG_PROCESSOR__BUNDLE_REFRESH_CHECK_INTERVAL
value: '120'
- name: AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL
value: '120'
- name: AIRFLOW__CORE__STORE_SERIALIZED_DAGS
value: 'True'
- name: AIRFLOW__CORE__MIN_SERIALIZED_DAG_UPDATE_INTERVAL
value: '60'
- name: AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL
value: '120'
- name: AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD
value: '86400'
```
One important point I observed is that whatever DAG is disappeared from UI if I check its detail on any of the airflow pod using command `airflow dags details DAG_ID` then I can see IsStale = true
Why DAGs are being marked as stale even after high value 86400 of AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD
### What you think should happen instead?
DAGs shouldn't disappear unless I delete these DAGs from the mounted volume.
### How to reproduce
Use same dag processor config and observer for few days. Intermittent issue.
### Operating System
Linux
### Versions of Apache Airflow Providers
eval_type_backport==0.2.2
apache-airflow-providers-databricks==6.4.0
apache-airflow-providers-mongo==4.2.1
apache-airflow-providers-git==0.0.2
apache-airflow-providers-standard==1.2.0
soda-core-spark-df==3.5.5
soda-core-spark[databricks]==3.5.5
soda-core-scientific==3.5.5
pymongo~=4.0.0
typing_extensions==4.13.2
paramiko<4
PyMuPDF~=1.26.5
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Official Apache Airflow Helm Chart
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the dag-processor entry point and the supplied Helm values.yaml configuration, then use `airflow dags details DAG_ID` to compare DAGs that remain visible with those reported as stale. Reproduce the intermittent disappearance over several days and identify why mounted DAGs become stale despite the configured threshold; done means valid DAGs remain visible without deleting their files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, python
- Domain
- data-engineering, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100