dask / dask/distributed

Tests leaking processes

Open
#1,597 13 comments 0 reactions 0 assignees View on GitHub
flaky test
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Thanks to https://github.com/dask/distributed/pull/1596 we can identify tests which leak child processes.

Here are the current results:
```
distributed/deploy/tests/test_adaptive.py::test_get_scale_up_kwargs leaked 1 processes:
- pid=23934, name='python', cmdline=('/home/antoine/miniconda3/envs/dask36/bin/python', '-s', '-c', "from multiprocessing.forkserver import main; main(27, 28, ['distributed', 'pkg_resources'], **{'sys_path': ['/home/antoine/distributed/distributed/deploy/tests', '/home/antoine/distributed', '/home/antoine/miniconda3/envs/dask36/bin', '/home/antoine/miniconda3/envs/dask36/lib/python36.zip', '/home/antoine/miniconda3/envs/dask36/lib/python3.6', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/lib-dynload', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages', '/home/antoine/dask', '/home/antoine/tornado', '/home/antoine/curio', '/home/antoine/cloudpickle', '/home/antoine/distributed', '/home/antoine/mtprof']})")
```
```
distributed/tests/test_client.py::test_client_async_before_loop_starts leaked 4 processes:
- pid=19583, name='python', cmdline=('/home/antoine/miniconda3/envs/dask36/bin/python', '-s', '-c', "from multiprocessing.forkserver import main; main(23, 25, ['distributed', 'pkg_resources'], **{'sys_path': ['/home/antoine/distributed/distributed/tests', '/home/antoine/distributed', '/home/antoine/miniconda3/envs/dask36/bin', '/home/antoine/miniconda3/envs/dask36/lib/python36.zip', '/home/antoine/miniconda3/envs/dask36/lib/python3.6', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/lib-dynload', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages', '/home/antoine/dask', '/home/antoine/tornado', '/home/antoine/curio', '/home/antoine/cloudpickle', '/home/antoine/distributed', '/home/antoine/mtprof', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages/IPython/extensions']})")
- pid=19586, name='python', cmdline=('/home/antoine/miniconda3/envs/dask36/bin/python', '-s', '-c', "from multiprocessing.forkserver import main; main(23, 25, ['distributed', 'pkg_resources'], **{'sys_path': ['/home/antoine/distributed/distributed/tests', '/home/antoine/distributed', '/home/antoine/miniconda3/envs/dask36/bin', '/home/antoine/miniconda3/envs/dask36/lib/python36.zip', '/home/antoine/miniconda3/envs/dask36/lib/python3.6', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/lib-dynload', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages', '/home/antoine/dask', '/home/antoine/tornado', '/home/antoine/curio', '/home/antoine/cloudpickle', '/home/antoine/distributed', '/home/antoine/mtprof', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages/IPython/extensions']})")
- pid=19589, name='python', cmdline=('/home/antoine/miniconda3/envs/dask36/bin/python', '-s', '-c', "from multiprocessing.forkserver import main; main(23, 25, ['distributed', 'pkg_resources'], **{'sys_path': ['/home/antoine/distributed/distributed/tests', '/home/antoine/distributed', '/home/antoine/miniconda3/envs/dask36/bin', '/home/antoine/miniconda3/envs/dask36/lib/python36.zip', '/home/antoine/miniconda3/envs/dask36/lib/python3.6', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/lib-dynload', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages', '/home/antoine/dask', '/home/antoine/tornado', '/home/antoine/curio', '/home/antoine/cloudpickle', '/home/antoine/distributed', '/home/antoine/mtprof', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages/IPython/extensions']})")
- pid=19592, name='python', cmdline=('/home/antoine/miniconda3/envs/dask36/bin/python', '-s', '-c', "from multiprocessing.forkserver import main; main(23, 25, ['distributed', 'pkg_resources'], **{'sys_path': ['/home/antoine/distributed/distributed/tests', '/home/antoine/distributed', '/home/antoine/miniconda3/envs/dask36/bin', '/home/antoine/miniconda3/envs/dask36/lib/python36.zip', '/home/antoine/miniconda3/envs/dask36/lib/python3.6', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/lib-dynload', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages', '/home/antoine/dask', '/home/antoine/tornado', '/home/antoine/curio', '/home/antoine/cloudpickle', '/home/antoine/distributed', '/home/antoine/mtprof', '/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages/IPython/extensions']})")
```
```
distributed/tests/test_utils.py::test_str_graph leaked 1 processes:
- pid=22719, name='py.test', cmdline=('/home/antoine/miniconda3/envs/dask36/bin/python', '/home/antoine/miniconda3/envs/dask36/bin/py.test', '-L', 'processes', '--leak-retries=0', 'distributed/tests/')
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.