dask / dask/distributed

Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name='<stderr>'> at interpreter shutdown, possibly due to daemon threads

Open
#7,617 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Please be concise with code posted. See guidelines below on how to provide a good bug report:
We use dask distributed in production, and at end of the run:

```
[2023-02-10 19:38:15,755] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,755] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,755] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,755] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,755] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,755] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,755] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,760] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,761] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,776] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,776] {subprocess.py:78} INFO - distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
[2023-02-10 19:38:15,792] {subprocess.py:78} INFO - distributed.scheduler - INFO - Scheduler closing...
[2023-02-10 19:38:15,798] {subprocess.py:78} INFO - distributed.scheduler - INFO - Scheduler closing all comms
[2023-02-10 19:38:15,847] {subprocess.py:78} INFO - 2023-02-10 19:38:15 INFO Notification code sent
[2023-02-10 19:38:15,929] {subprocess.py:78} INFO - /duacsng/core/src/duacsng/core/dnglogging.py:31 - INFO - duacsng_params common.notification_emailⓩduacs_pro@groupcls.com
[2023-02-10 19:38:15,991] {subprocess.py:78} INFO - 2023-02-10 19:38:15 INFO duacsng_params common.smtp_serverⓩmail.cls.fr
[2023-02-10 19:38:15,992] {subprocess.py:78} INFO - /duacsng/core/src/duacsng/core/dnglogging.py:31 - INFO - duacsng_params common.smtp_serverⓩmail.cls.fr
[2023-02-10 19:38:16,018] {subprocess.py:78} INFO - distributed.nanny - INFO - Worker process 25591 was killed by signal 15
[2023-02-10 19:38:16,020] {subprocess.py:78} INFO - distributed.nanny - INFO - Worker process 25597 was killed by signal 15
[2023-02-10 19:38:16,024] {subprocess.py:78} INFO - distributed.nanny - INFO - Worker process 25610 was killed by signal 15
[2023-02-10 19:38:16,037] {subprocess.py:78} INFO - distributed.nanny - INFO - Worker process 25573 was killed by signal 15
[2023-02-10 19:38:16,039] {subprocess.py:78} INFO - distributed.nanny - INFO - Worker process 25635 was killed by signal 15
[2023-02-10 19:38:16,044] {subprocess.py:78} INFO - distributed.process - INFO - reaping stray process
[2023-02-10 19:38:16,045] {subprocess.py:78} INFO - distributed.process - INFO - reaping stray process
[2023-02-10 19:38:16,045] {subprocess.py:78} INFO - distributed.process - INFO - reaping stray process
[2023-02-10 19:38:16,045] {subprocess.py:78} INFO - distributed.process - INFO - reaping stray process
[2023-02-10 19:38:16,048] {subprocess.py:78} INFO - distributed.process - INFO - reaping stray process
[2023-02-10 19:38:16,048] {subprocess.py:78} INFO - distributed.process - INFO - reaping stray process
[2023-02-10 19:38:16,057] {subprocess.py:78} INFO - distributed.nanny - INFO - Worker process 25622 was killed by signal 15
[2023-02-10 19:38:16,059] {subprocess.py:78} INFO - distributed.nanny - INFO - Worker process 25584 was killed by signal 15
[2023-02-10 19:38:16,063] {subprocess.py:78} INFO - Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
[2023-02-10 19:38:16,063] {subprocess.py:78} INFO - Traceback (most recent call last):
[2023-02-10 19:38:16,063] {subprocess.py:78} INFO - File "/opt/conda/envs/octant/lib/python3.9/threading.py", line 973, in _bootstrap_inner
[2023-02-10 19:38:16,064] {subprocess.py:78} INFO - distributed.nanny - INFO - Worker process 25616 was killed by signal 15
[2023-02-10 19:38:16,064] {subprocess.py:78} INFO - distributed.nanny - INFO - Worker process 25579 was killed by signal 15
[2023-02-10 19:38:16,065] {subprocess.py:78} INFO - self.run()
[2023-02-10 19:38:16,065] {subprocess.py:78} INFO - Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join:
[2023-02-10 19:38:16,065] {subprocess.py:78} INFO - File "/opt/conda/envs/octant/lib/python3.9/threading.py", line 910, in run
[2023-02-10 19:38:16,066] {subprocess.py:78} INFO - Traceback (most recent call last):
[2023-02-10 19:38:17,066] {subprocess.py:78} INFO - Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name=''> at interpreter shutdown, possibly due to daemon threads
[2023-02-10 19:38:17,067] {subprocess.py:78} INFO - Python runtime state: finalizing (tstate=0x55f67e9fb800)
[2023-02-10 19:38:17,067] {subprocess.py:78} INFO -
[2023-02-10 19:38:17,067] {subprocess.py:78} INFO - Current thread 0x00007f9fb3ea1700 (most recent call first):
[2023-02-10 19:38:17,067] {subprocess.py:78} INFO -
[2023-02-10 19:38:20,217] {subprocess.py:82} INFO - Command exited with return code -6
[2023-02-10 19:38:21,296] {taskinstance.py:1462} ERROR - Task failed with exception
```

This error happened only once for us, we would be happy if you could provide us an explanation of what happened.

**Environment**:

- Dask version: 2021.11.2
- Python version: '3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:46) \n[GCC 9.4.0]'
- Operating System: Linux: xenial
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.