Allow workers to cancel running task?
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Currently, if a task is already running on a thread of a worker, it will always run to completion/exception even if the work is no longer required (e.g., the future was explicitly cancelled). We usually say "you cannot interrupt a python thread", so Dask has no option here. However, you [can](https://blog.finxter.com/how-to-kill-a-thread-in-python/#Method_4_Using_ctypes_To_Raise_Exceptions_In_A_Thread) set the thread state to exception using `ctypes.pythonapi.PyThreadState_SetAsyncExc`. Yes, this is a C-api call, but it is public and stable. Is this worth pursuing? It would allow not only early cancellation of unneeded work, but "interrupts" and timeouts from the worker itself, perhaps even better debugging if it can break deadlocks with a useful traceback.
(Note that the setting of an exception only has effect at the next python bytecode instruction, so some long-running C-level function or even `wait(big)` will not raise immediately)
Contributor guide
Assessment
This issue has not been assessed yet.