local stop logic - clean/kill docker containers for failed local experiment running
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
## *Who* is the bug affecting?
The local running users, especially testing new changes in code. Probably, it also applies to Google Cloud based running.
Also, related to implement end-to-end local test running (#465)
.
## *When* does this occur?
When the experiment running has some problems in the middle and not gracefully stopped, the runners docker containers are still running until the preset `max_total_time` reaches.
## *Where* on the platform does it happen?
Linux 5.2.17-1rodete3-amd64, CPU: 16 cores, RAM: 62.9 G
Docker version 19.03.5, build 633a0ea838
## *How* do we replicate the issue?
Run an experiment, and wait for the scheduler runs trial runners docker containers. Like this:
```
...
INFO:root:Done building fuzzer benchmarks.
INFO:root:Starting scheduler.
INFO:root:Finding trials to schedule.
INFO:root:Starting trials.
INFO:root:Start measure_loop.
INFO:root:Start trial 43.
INFO:root:Start trial 52.
INFO:root:Start trial 53.
INFO:root:Start trial 45.
INFO:root:Start trial 33.
INFO:root:Start trial 28.
INFO:root:Start trial 34.
...
```
Then type `Ctrl+C` several times to kill the process and exit to the bash for dispatch-container:
```
...
KeyboardInterrupt
^CException ignored in:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/threading.py", line 1307, in _shutdown
lock.acquire()
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/util.py", line 277, in _run_finalizers
finalizer()
File "/usr/local/lib/python3.7/multiprocessing/util.py", line 201, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 581, in _terminate_pool
cls._help_stuff_finish(inqueue, task_handler, len(pool))
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 566, in _help_stuff_finish
inqueue._rlock.acquire()
KeyboardInterrupt
^C
root@c43309f9a9d9:/work# /usr/local/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 4 leaked semaphores to clean up at shutdown
len(cache))
^C
root@c43309f9a9d9:/work#
```
Then type `exit` to return back to the bash terminal:
```
root@c43309f9a9d9:/work# exit
exit
Traceback (most recent call last):
File "experiment/run_experiment.py", line 555, in
sys.exit(main())
File "experiment/run_experiment.py", line 550, in main
args.experiment_config)
File "/usr/local/home/username/fuzzbench/experiment/stop_experiment.py", line 33, in stop_experiment
'Local experiment stop logic is not implemented.')
NotImplementedError: Local experiment stop logic is not implemented.
(.venv) hostname@username:~/fuzzbench$
```
Then type `docker ps` and there exists unfinished docker containers still running, which are those trial runners.
```
(.venv) hostname@username:~/fuzzbench$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8b6eee9a0b69 gcr.io/fuzzbench/runners/fairfuzz/zlib_zlib_uncompress_fuzzer "/bin/sh -c $ROOT_DI…" 6 minutes ago Up 5 minutes serene_elbakyan
c96a2142b001 gcr.io/fuzzbench/runners/aflfast/jsoncpp_jsoncpp_fuzzer "/bin/sh -c $ROOT_DI…" 6 minutes ago Up 5 minutes gracious_euler
...
```
After around the time for `max_total_time`, type `docker ps` again and those docker containers are gone.
## Expected behavior (i.e. solution)
Three possible expectations:
(1) After we type `Ctrl+C`, those started runners, related to this experiment, should be killed.
(2) After we exit from the dispatcher-container, those started runners, related to this experiment, should be killed.
(3) Provide a script in `Makefile`, for example `make clean-containers -e ` to kill them. We can enhance/specify the name for trial runners by including `experiment_name` term and search for them to kill.
I propose to do (2) and (3) together. The way to do (2) is in `experiment/stop_experiemnt.py`, which also solves the `NotImplementError` there:
https://github.com/google/fuzzbench/blob/7805093436f36ddaef155da8b045da2b57ec02b2/experiment/stop_experiment.py#L27-L33
## Other Comments
For a normal experiment running, this will not happen, since dispatcher-container will not exit until all trials are finished running. This is happening for a failed try.
This may also be a requirement for implementing end-to-end local test. (#465) When an end-to-end test fails, we want no side-effects.
Related to #251 and #444.
Contributor guide
Assessment
This issue has not been assessed yet.