google / google/ml-compiler-opt

cancellation_manager.pause_all_work() is insufficient to pause work

Open
#116 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
798
Forks
123
Avg merge
55m
Merged PRs (30d)
4

Description

It is possible for work to continue running if no processes are registered on the cancellation manager when pause_all_work() is called.

Possible solutions:
Also send SIGSTOP to the worker python process- however this has a race condition: pause_all_work -> new work scheduled -> pause python -> work is still being done. pausing python first would prevent pause_all_work from running so that's not an option.

Alternatively, have cancellation manager send SIGSTOP upon process registration when paused. This should work, I think, and is probably the easiest solution.

to verify: add a time_sleep right under .pause_children() in train_locally and look at the cpu graph

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.