apache / apache/airflow

DagBag Processing SIGSEGV causes runaway tasks with LocalExecutor

Open
#57,174 9 comments 0 reactions 0 assignees View on GitHub
area:core area:Executors-core good first issue kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

3.1.0

### If "Other Airflow 2/3 version" selected, which one?

_No response_

### What happened?

Using Airflow 3.1.0 - `airflow standalone` with the LocalExecutor on MacOS 14 - M3 ARM

I've noticed that occasionally tasks that are dynamically created (with `.expand()`, I don't think this is the issue fyi, just including), a subset of the them can runway, and will not fail even with a `execution_timeout` set. It completely disregards the timeout after this error is found in the logs:
```
ERROR - Received SIGSEGV signal while processing . source=airflow.models.dagbag.DagBag loc=dagbag.py:375
```

Here's some screenshots from a manually failed run, note the duration of 30 minutes even though a max timeout of 3 minutes:

Image

Log from the Task:
Image

Dag definition (linting errors are unrelated to this issue and are in a different task.):
Image

---

I have a hunch this happens when the DAG Processor starts running at the same time as all of the tasks are dynamically created, but I truthfully have no clue.

I don't really have an issue with tasks like this failing on standalone due to a perfect storm of conditions (and non-intended OSes) as I only use it for local testing, but I would like the tasks to not run away.

---

Possibly Related Ticket:
https://github.com/apache/airflow/issues/47011
https://github.com/apache/airflow/issues/55838
https://github.com/apache/airflow/issues/50303
https://github.com/apache/airflow/discussions/35967

### What you think should happen instead?

_No response_

### How to reproduce

No clue, besides try casting a wide net with many dynamically created tasks in the chance that this error bubbles up.
I've included my DAG code besides my custom BoxHook in case this helps.

### Operating System

MacOS 14 - M3 ARM

### Versions of Apache Airflow Providers

apache-airflow-providers-google==18.0.0
apache-airflow-providers-standard==1.8.0

### Deployment

Other

### Deployment details

Local Standalone

### Anything else?

I think it bubbles up in the DAG run about 20% of the time.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start at airflow.models.dagbag.DagBag and dagbag.py:375, then review the linked issues for related SIGSEGV behavior. Try reproducing with Airflow 3.1.0, LocalExecutor, standalone mode, MacOS 14 on M3, and many dynamically expanded tasks; done means identifying why tasks outlive execution_timeout and preventing runaway execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.