apache / apache/airflow

Local Executor test_execution[unlimited] seems to randomly fail

Open
#45,683 2 comments 0 reactions 0 assignees View on GitHub
area:CI area:Executors-core
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

Recently test_execution started to fail with:

```
FAILED tests/executors/test_local_executor.py::TestLocalExecutor::test_execution[unlimited] - Failed: Timeout >60.0s
```

Stacktrace:

```python
_________________ TestLocalExecutor.test_execution[unlimited] __________________
tests/executors/test_local_executor.py:129: in test_execution
self._test_execute(parallelism=parallelism)
/usr/local/lib/python3.10/unittest/mock.py:1379: in patched
return func(*newargs, **newkeywargs)
tests/executors/test_local_executor.py:106: in _test_execute
executor.end()
airflow/executors/local_executor.py:234: in end
proc.join()
/usr/local/lib/python3.10/multiprocessing/process.py:149: in join
res = self._popen.wait(timeout)
/usr/local/lib/python3.10/multiprocessing/popen_fork.py:43: in wait
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
/usr/local/lib/python3.10/multiprocessing/popen_fork.py:27: in poll
pid, sts = os.waitpid(self.pid, flag)
E Failed: Timeout >60.0s
----------------------------- Captured stdout call -----------------------------
[2025-01-15T02:12:46.747+0000] {local_executor.py:63} INFO - Worker starting up pid=91
[2025-01-15T02:12:46.761+0000] {local_executor.py:63} INFO - Worker starting up pid=92
[2025-01-15T02:12:46.769+0000] {local_executor.py:99} ERROR - uhoh
Traceback (most recent call last):
File "/opt/airflow/airflow/executors/local_executor.py", line 95, in _run_worker
_execute_work(log, workload)
File "/opt/airflow/airflow/executors/local_executor.py", line 116, in _execute_work
supervise(
File "/usr/local/lib/python3.10/unittest/mock.py", line 1114, in __call__
return self._mock_call(*args, **kwargs)
File "/usr/local/lib/python3.10/unittest/mock.py", line 1118, in _mock_call
return self._execute_mock_call(*args, **kwargs)
File "/usr/local/lib/python3.10/unittest/mock.py", line 1179, in _execute_mock_call
result = effect(*args, **kwargs)
File "/opt/airflow/tests/executors/test_local_executor.py", line 74, in fake_supervise
raise RuntimeError("fake failure")
RuntimeError: fake failure
[2025-01-15T02:12:46.769+0000] {local_executor.py:219} INFO - Shutting down LocalExecutor; waiting for running tasks to finish. Signal again if you don't want to wait.
[2025-01-15T02:12:46.778+0000] {local_executor.py:63} INFO - Worker starting up pid=93
```

Example failure: https://github.com/apache/airflow/actions/runs/12779948905/job/35625787707#step:6:7344

Contributor guide

Open the contributing guide

Research direction

Start with tests/executors/test_local_executor.py, especially TestLocalExecutor.test_execution[unlimited] and _test_execute, then inspect the worker shutdown path in airflow/executors/local_executor.py, including end() and _run_worker. Reproduce the test repeatedly and trace the mocked fake_supervise failure. Done means the test no longer randomly hangs or exceeds the 60-second timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.