Dask unmanaged memory terminate worker causing error: ERROR:asyncio:Future exception was never retrieved
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I have worked with large data that exceed RAM memory. When occupied RAM exceeded terminate threshold, Dask was trying to restart the worker and causing the error as below. I was adding del for known variables and gc.collect() also doesn't help much.
My environment includes Windows 10, python 3.10. 16GB RAM, and with 20GB data.
Any help/idea would be much appreciated!
`WARNING:distributed.nanny.memory:Worker tcp://127.0.0.1:60050 (pid=14536) is slow to terminate; trying again
ERROR:asyncio:Future exception was never retrieved
future:
Traceback (most recent call last):
File "C:\Users\hung.bui\AppData\Local\Programs\HCLTech\AION\2.7.0.7\aion-venv\lib\site-packages\distributed\process.py", line 55, in _call_and_set_future
res = func(*args, **kwargs)
File "C:\Users\hung.bui\AppData\Local\Programs\HCLTech\AION\2.7.0.7\python\lib\multiprocessing\process.py", line 140, in kill
self._popen.kill()
File "C:\Users\hung.bui\AppData\Local\Programs\HCLTech\AION\2.7.0.7\python\lib\multiprocessing\popen_spawn_win32.py", line 123, in terminate
_winapi.TerminateProcess(int(self._handle), TERMINATE)
PermissionError: [WinError 5] Access is denied
ERROR:asyncio:Future exception was never retrieved
future:
Traceback (most recent call last):
File "C:\Users\hung.bui\AppData\Local\Programs\HCLTech\AION\2.7.0.7\aion-venv\lib\site-packages\distributed\process.py", line 55, in _call_and_set_future
res = func(*args, **kwargs)
File "C:\Users\hung.bui\AppData\Local\Programs\HCLTech\AION\2.7.0.7\python\lib\multiprocessing\process.py", line 140, in kill
self._popen.kill()
File "C:\Users\hung.bui\AppData\Local\Programs\HCLTech\AION\2.7.0.7\python\lib\multiprocessing\popen_spawn_win32.py", line 123, in terminate
_winapi.TerminateProcess(int(self._handle), TERMINATE)
PermissionError: [WinError 5] Access is denied
ERROR:asyncio:Future exception was never retrieved
future:
Traceback (most recent call last):
File "C:\Users\hung.bui\AppData\Local\Programs\HCLTech\AION\2.7.0.7\aion-venv\lib\site-packages\distributed\process.py", line 55, in _call_and_set_future
res = func(*args, **kwargs)
File "C:\Users\hung.bui\AppData\Local\Programs\HCLTech\AION\2.7.0.7\python\lib\multiprocessing\process.py", line 140, in kill
self._popen.kill()
File "C:\Users\hung.bui\AppData\Local\Programs\HCLTech\AION\2.7.0.7\python\lib\multiprocessing\popen_spawn_win32.py", line 123, in terminate
_winapi.TerminateProcess(int(self._handle), TERMINATE)
PermissionError: [WinError 5] Access is denied
`
Contributor guide
Assessment
This issue has not been assessed yet.