dask / dask/distributed

Potentially flaky `test_stress` and `test_ucx_protocol[1234]` UCX tests

Open
#5,542 0 comments 0 reactions 0 assignees View on GitHub
flaky test
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I could not reproduce those locally, but I've seen both `test_stress` and `test_ucx_protocol[1234]` fail in https://gpuci.gpuopenanalytics.com/job/dask/job/distributed/job/prb/job/distributed-prb/900/CUDA_VER=11.2,LINUX_VER=ubuntu18.04,PYTHON_VER=3.8,RAPIDS_VER=21.12/console , details follow.

```python
20:48:57 _______________________ ERROR at teardown of test_stress _______________________
20:48:57

20:48:57 scope = 'function'
20:48:57

20:48:57 @pytest.fixture()
20:48:57 def event_loop(scope="function"):
20:48:57 loop = asyncio.new_event_loop()
20:48:57 loop.set_exception_handler(handle_exception)
20:48:57 ucp.reset()
20:48:57 yield loop
20:48:57 > ucp.reset()
20:48:57

20:48:57 distributed/comm/tests/test_ucx.py:38:
20:48:57 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
20:48:57

20:48:57 def reset():
20:48:57 """Resets the UCX library by shutting down all of UCX.
20:48:57
20:48:57 The library is initiated at next API call.
20:48:57 """
20:48:57 global _ctx
20:48:57 if _ctx is not None:
20:48:57 weakref_ctx = weakref.ref(_ctx)
20:48:57 _ctx = None
20:48:57 gc.collect()
20:48:57 if weakref_ctx() is not None:
20:48:57 msg = (
20:48:57 "Trying to reset UCX but not all Endpoints and/or Listeners "
20:48:57 "are closed(). The following objects are still referencing "
20:48:57 "ApplicationContext: "
20:48:57 )
20:48:57 for o in gc.get_referrers(weakref_ctx()):
20:48:57 msg += "\n %s" % str(o)
20:48:57 > raise UCXError(msg)
20:48:57 E ucp.exceptions.UCXError: Trying to reset UCX but not all Endpoints and/or Listeners are closed(). The following objects are still referencing ApplicationContext:
20:48:57 E
20:48:57 E
20:48:57 E
20:48:57 E
20:48:57 E
20:48:57 E
20:48:57 E
20:48:57 E
20:48:57

20:48:57 /opt/conda/envs/dask/lib/python3.8/site-packages/ucp/core.py:938: UCXError
20:48:57 _________________ ERROR at teardown of test_ucx_protocol[1234] _________________
20:48:57

20:48:57 scope = 'function'
20:48:57

20:48:57 @pytest.fixture()
20:48:57 def event_loop(scope="function"):
20:48:57 loop = asyncio.new_event_loop()
20:48:57 loop.set_exception_handler(handle_exception)
20:48:57 ucp.reset()
20:48:57 yield loop
20:48:57 > ucp.reset()
20:48:57

20:48:57 distributed/comm/tests/test_ucx.py:38:
20:48:57 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
20:48:57

20:48:57 def reset():
20:48:57 """Resets the UCX library by shutting down all of UCX.
20:48:57
20:48:57 The library is initiated at next API call.
20:48:57 """
20:48:57 global _ctx
20:48:57 if _ctx is not None:
20:48:57 weakref_ctx = weakref.ref(_ctx)
20:48:57 _ctx = None
20:48:57 gc.collect()
20:48:57 if weakref_ctx() is not None:
20:48:57 msg = (
20:48:57 "Trying to reset UCX but not all Endpoints and/or Listeners "
20:48:57 "are closed(). The following objects are still referencing "
20:48:57 "ApplicationContext: "
20:48:57 )
20:48:57 for o in gc.get_referrers(weakref_ctx()):
20:48:57 msg += "\n %s" % str(o)
20:48:57 > raise UCXError(msg)
20:48:57 E ucp.exceptions.UCXError: Trying to reset UCX but not all Endpoints and/or Listeners are closed(). The following objects are still referencing ApplicationContext:
20:48:57 E
20:48:57 E
20:48:57

20:48:57 /opt/conda/envs/dask/lib/python3.8/site-packages/ucp/core.py:938: UCXError
```

Please post details here should this be encountered again.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.