google / google/ml-compiler-opt
cancellation_manager.pause_all_work() is insufficient to pause work
- Dominant language
- Python
- Stars
- 798
- Forks
- 123
- Avg merge
- 55m
- Merged PRs (30d)
- 4
Description
It is possible for work to continue running if no processes are registered on the cancellation manager when pause_all_work() is called.
Possible solutions:
Also send SIGSTOP to the worker python process- however this has a race condition: pause_all_work -> new work scheduled -> pause python -> work is still being done. pausing python first would prevent pause_all_work from running so that's not an option.
Alternatively, have cancellation manager send SIGSTOP upon process registration when paused. This should work, I think, and is probably the easiest solution.
to verify: add a time_sleep right under .pause_children() in train_locally and look at the cpu graph
Contributor guide
Assessment
This issue has not been assessed yet.