DagBag Processing SIGSEGV causes runaway tasks with LocalExecutor
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
3.1.0
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
Using Airflow 3.1.0 - `airflow standalone` with the LocalExecutor on MacOS 14 - M3 ARM
I've noticed that occasionally tasks that are dynamically created (with `.expand()`, I don't think this is the issue fyi, just including), a subset of the them can runway, and will not fail even with a `execution_timeout` set. It completely disregards the timeout after this error is found in the logs:
```
ERROR - Received SIGSEGV signal while processing . source=airflow.models.dagbag.DagBag loc=dagbag.py:375
```
Here's some screenshots from a manually failed run, note the duration of 30 minutes even though a max timeout of 3 minutes:
Log from the Task:
Dag definition (linting errors are unrelated to this issue and are in a different task.):
---
I have a hunch this happens when the DAG Processor starts running at the same time as all of the tasks are dynamically created, but I truthfully have no clue.
I don't really have an issue with tasks like this failing on standalone due to a perfect storm of conditions (and non-intended OSes) as I only use it for local testing, but I would like the tasks to not run away.
---
Possibly Related Ticket:
https://github.com/apache/airflow/issues/47011
https://github.com/apache/airflow/issues/55838
https://github.com/apache/airflow/issues/50303
https://github.com/apache/airflow/discussions/35967
### What you think should happen instead?
_No response_
### How to reproduce
No clue, besides try casting a wide net with many dynamically created tasks in the chance that this error bubbles up.
I've included my DAG code besides my custom BoxHook in case this helps.
### Operating System
MacOS 14 - M3 ARM
### Versions of Apache Airflow Providers
apache-airflow-providers-google==18.0.0
apache-airflow-providers-standard==1.8.0
### Deployment
Other
### Deployment details
Local Standalone
### Anything else?
I think it bubbles up in the DAG run about 20% of the time.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start at airflow.models.dagbag.DagBag and dagbag.py:375, then review the linked issues for related SIGSEGV behavior. Try reproducing with Airflow 3.1.0, LocalExecutor, standalone mode, MacOS 14 on M3, and many dynamically expanded tasks; done means identifying why tasks outlive execution_timeout and preventing runaway execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100