Local Executor test_execution[unlimited] seems to randomly fail
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
Recently test_execution started to fail with:
```
FAILED tests/executors/test_local_executor.py::TestLocalExecutor::test_execution[unlimited] - Failed: Timeout >60.0s
```
Stacktrace:
```python
_________________ TestLocalExecutor.test_execution[unlimited] __________________
tests/executors/test_local_executor.py:129: in test_execution
self._test_execute(parallelism=parallelism)
/usr/local/lib/python3.10/unittest/mock.py:1379: in patched
return func(*newargs, **newkeywargs)
tests/executors/test_local_executor.py:106: in _test_execute
executor.end()
airflow/executors/local_executor.py:234: in end
proc.join()
/usr/local/lib/python3.10/multiprocessing/process.py:149: in join
res = self._popen.wait(timeout)
/usr/local/lib/python3.10/multiprocessing/popen_fork.py:43: in wait
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
/usr/local/lib/python3.10/multiprocessing/popen_fork.py:27: in poll
pid, sts = os.waitpid(self.pid, flag)
E Failed: Timeout >60.0s
----------------------------- Captured stdout call -----------------------------
[2025-01-15T02:12:46.747+0000] {local_executor.py:63} INFO - Worker starting up pid=91
[2025-01-15T02:12:46.761+0000] {local_executor.py:63} INFO - Worker starting up pid=92
[2025-01-15T02:12:46.769+0000] {local_executor.py:99} ERROR - uhoh
Traceback (most recent call last):
File "/opt/airflow/airflow/executors/local_executor.py", line 95, in _run_worker
_execute_work(log, workload)
File "/opt/airflow/airflow/executors/local_executor.py", line 116, in _execute_work
supervise(
File "/usr/local/lib/python3.10/unittest/mock.py", line 1114, in __call__
return self._mock_call(*args, **kwargs)
File "/usr/local/lib/python3.10/unittest/mock.py", line 1118, in _mock_call
return self._execute_mock_call(*args, **kwargs)
File "/usr/local/lib/python3.10/unittest/mock.py", line 1179, in _execute_mock_call
result = effect(*args, **kwargs)
File "/opt/airflow/tests/executors/test_local_executor.py", line 74, in fake_supervise
raise RuntimeError("fake failure")
RuntimeError: fake failure
[2025-01-15T02:12:46.769+0000] {local_executor.py:219} INFO - Shutting down LocalExecutor; waiting for running tasks to finish. Signal again if you don't want to wait.
[2025-01-15T02:12:46.778+0000] {local_executor.py:63} INFO - Worker starting up pid=93
```
Example failure: https://github.com/apache/airflow/actions/runs/12779948905/job/35625787707#step:6:7344
Contributor guide
Research direction
Start with tests/executors/test_local_executor.py, especially TestLocalExecutor.test_execution[unlimited] and _test_execute, then inspect the worker shutdown path in airflow/executors/local_executor.py, including end() and _run_worker. Reproduce the test repeatedly and trace the mocked fake_supervise failure. Done means the test no longer randomly hangs or exceeds the 60-second timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100