rapidsai / rapidsai/dask-upstream-testing

Nightly faliures

Open
#101 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
2
Forks
7
Avg merge
2m
Merged PRs (30d)
1

Description

CuML

Some timeouts, e.g in packages/cuml/python/cuml/tests/dask/test_dask_base.py::test_get_combined_model

Error: test_get_combined_model[True-data_size0-LinearRegression-float32]

Failed: Timeout (>120.0s) from pytest-timeout.
packages/cuml/python/cuml/tests/dask/test_dask_base.py::test_get_combined_model[True-data_size0-LinearRegression-float32] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~ Stack of AsyncProcess Dask Worker process (from Nanny) watch process join (138015837894336) ~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 250, in _watch_process
    process.join()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 43, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
~ Stack of AsyncProcess Dask Worker process (from Nanny) watch message queue (138015175337664) ~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 230, in _watch_message_queue
    msg = q.get()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/queue.py", line 171, in get
    self.not_empty.wait()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~ Stack of QueueFeederThread (138015183730368) ~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py", line 251, in _feed
    nwait()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~~~~~~ Stack of Profile (138015854679744) ~~~~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/profile.py", line 370, in _watch
    sleep(interval)
~~~~~~~~~~~~~~~~~~~~~~ Stack of IO loop (138015846287040) ~~~~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 466, in wrapper
    target()
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 574, in run_loop
    asyncio_run(amain(), loop_factory=get_loop_factory())
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 678, in run_until_complete
    self.run_forever()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 645, in run_forever
    self._run_once()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 1961, in _run_once
    event_list = self._selector.select(timeout)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
~~~~~~~~~~~~~~~~~~ Stack of Dask-Offload_0 (138014556595904) ~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 90, in _worker
    work_item = work_queue.get(block=True)
~~~~~~~~~~~~~~~ Stack of Thread-1 (run_server) (138025835161280) ~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pytest_rerunfailures.py", line 474, in run_server
    conn, _ = self.sock.accept()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/socket.py", line 295, in accept
    fd, addr = self._accept()
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
FAILED                                                                   [  4%]
Error: test_get_combined_model[True-data_size0-LinearRegression-float64]

Failed: Timeout (>120.0s) from pytest-timeout.
packages/cuml/python/cuml/tests/dask/test_dask_base.py::test_get_combined_model[True-data_size0-LinearRegression-float64] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~ Stack of AsyncProcess Dask Worker process (from Nanny) watch process join (138015837894336) ~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 250, in _watch_process
    process.join()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 43, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
~ Stack of AsyncProcess Dask Worker process (from Nanny) watch message queue (138015175337664) ~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 230, in _watch_message_queue
    msg = q.get()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/queue.py", line 171, in get
    self.not_empty.wait()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~ Stack of QueueFeederThread (138015183730368) ~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py", line 251, in _feed
    nwait()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~~~~~~ Stack of Profile (138015854679744) ~~~~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/profile.py", line 370, in _watch
    sleep(interval)
~~~~~~~~~~~~~~~~~~~~~~ Stack of IO loop (138015846287040) ~~~~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 466, in wrapper
    target()
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 574, in run_loop
    asyncio_run(amain(), loop_factory=get_loop_factory())
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 678, in run_until_complete
    self.run_forever()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 645, in run_forever
    self._run_once()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 1961, in _run_once
    event_list = self._selector.select(timeout)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
~~~~~~~~~~~~~~~~~~ Stack of Dask-Offload_0 (138014556595904) ~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 90, in _worker
    work_item = work_queue.get(block=True)
~~~~~~~~~~~~~~~ Stack of Thread-1 (run_server) (138025835161280) ~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pytest_rerunfailures.py", line 474, in run_server
    conn, _ = self.sock.accept()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/socket.py", line 295, in accept
    fd, addr = self._accept()
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Warning: coroutine 'Client._run' was never awaited
Warning: coroutine 'Client._run_on_scheduler' was never awaited
FAILED                                                                   [  4%]

And a segfault

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Warning: Exception ignored in: <function Comms.__del__ at 0x7d889f12cfe0>

Traceback (most recent call last):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/raft_dask/common/comms.py", line 124, in __del__
    self.destroy()
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/raft_dask/common/comms.py", line 215, in destroy
    self.client.run(
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/client.py", line 3230, in run
    return self.sync(
           ^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 379, in sync
    return sync(
           ^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 455, in sync
    raise error
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 429, in f
    result = yield future
             ^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/tornado/gen.py", line 783, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/client.py", line 3106, in _run
    responses = await self.scheduler.broadcast(
                      ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'broadcast'

Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
FAILED                                                                   [  5%]
Fatal Python error: Segmentation fault

Thread 0x00007d8650fdb6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 27 in poll
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 43 in wait
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 149 in join
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 250 in _watch_process
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d8629fff6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355 in wait
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/queue.py", line 171 in get
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 230 in _watch_message_queue
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d86297fe6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355 in wait
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py", line 251 in _feed
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d8651fdd6c0 (most recent call first):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/profile.py", line 370 in _watch
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d86517dc6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/selectors.py", line 468 in select
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 1961 in _run_once
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 645 in run_forever
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 678 in run_until_complete
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195 in run
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 574 in run_loop
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 466 in wrapper
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d86049ea6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d88a4dfe6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/socket.py", line 295 in accept
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pytest_rerunfailures.py", line 474 in run_server
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Current thread 0x00007d8a34558740 (most recent call first):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/internals/outputs.py", line 433 in inner
  File "/__w/dask-upstream-testing/dask-upstream-testing/packages/cuml/python/cuml/tests/dask/test_dask_base.py", line 124 in test_regressor_sg_train_mg_predict
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/python.py", line 1720 in runtest
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 179 in pytest_runtest_call
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 245 in <lambda>
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 353 in from_call
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 244 in call_and_report
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 137 in runtestprotocol
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/main.py", line 396 in pytest_runtestloop
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/main.py", line 372 in _main
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 199 in main
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 223 in console_main
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/bin/pytest", line 10 in <module>

Extension modules: markupsafe._speedups, sklearn.__check_build._check_build, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, cython.cimports.libc.math, scipy._lib._ccallback_c, charset_normalizer.md, charset_normalizer.cd, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, psutil._psutil_linux, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, _cyutility, sklearn._cyutility, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, cupy_backends.cuda._softlink, cupy_backends.cuda.api._runtime_enum, cupy_backends.cuda.api.runtime, cupy._util, cupy.cuda.device, cupy.cuda.memory_hook, cupy_backends.cuda.stream, cupy.cuda.graph, cupy.cuda.stream, cupy_backends.cuda.api._driver_enum, cupy_backends.cuda.api.driver, cupy.cuda.memory, cupy._core.internal, cupy._core._carray, cupy._core._dtype, cupy._core._scalar, cupy.cuda.texture, cupy.cuda.function, cupy_backends.cuda.libs.nvrtc, cupy.cuda.pinned_memory, cupy.cuda.common, cupy.cuda.cub, cupy_backends.cuda.libs.nvtx, cupy.cuda.thrust, cupy._core._accelerator, cupy._core._memory_range, cupy._core._fusion_thread_local, cupy._core._kernel, cupy._core._routines_manipulation, cupy._core._routines_binary, cupy._core._optimize_config, cupy._core._cub_reduction, cupy._core._reduction, cupy._core._routines_math, cupy._core._routines_indexing, cupy._core._routines_linalg, cupy._core._routines_logic, cupy._core._routines_sorting, cupy._core._routines_statistics, cupy._core.dlpack, cupy._core.flags, cupy._core.core, cupy._core._fusion_variable, cupy._core._fusion_trace, cupy._core._fusion_kernel, cupy._core.new_fusion, cupy._core.fusion, cupy._core.raw, cupy.fft._cache, cupy.fft._callback, cupy.random._generator_api, cupy.random._bit_generator, cupy.lib._polynomial, numba.cuda.types.abstract.itertools, numba.cuda.types.functions.itertools, numba.cuda.cext._typeconv, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda.core.system._system, cuda.bindings._internal.utils, cuda.bindings._internal.nvml, cuda.bindings.cynvml, cuda.bindings.nvml, cuda.core.system._nvml_context, cuda.core.cu12.system._nvml_context, cuda.core.cu12.system._system, cuda.core.system._device, cuda.core.cu12.system._device, cuda.core.system._system_events, cuda.core.cu12.system._system_events, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings._internal.nvvm, cuda.bindings.cynvvm, cuda.bindings._internal.nvjitlink, cuda.bindings.cynvjitlink, cuda.bindings.nvrtc, cuda.bindings.nvvm, cuda.bindings.nvjitlink, cuda.core._utils.cuda_utils, cuda.core._layout, cuda.core._resource_handles, cuda.core._dlpack, cuda.core._context, cuda.core._event, cuda.core._stream, cuda.core._memory._ipc, cuda.core._memory._memory_pool, cuda.core._memory._device_memory_resource, cuda.core._memory._pinned_memory_resource, cuda.core._device, cuda.core._memory._buffer, cuda.core._memory._graph_memory_resource, cuda.core._memory._managed_memory_resource, cuda.core._utils.version, cuda.core._memoryview, cuda.core._graphics, cuda.core._launch_config, cuda.core._tensor_map, cuda.core._kernel_arg_handler, cuda.core._module, cuda.core._launcher, cuda.core._linker, cuda.core._program, cuda.core.graph._utils, cuda.core.graph._graph_builder, cuda.core.graph._subclasses, cuda.core.graph._adjacency_set_proxy, cuda.core.graph._graph_node, cuda.core.graph._graph_def, numba.cuda.cext._helperlib, numba.cuda.cext.mviewbuf, numba.cuda.cext._dispatcher, rmm.pylibrmm.stream, rmm.pylibrmm.utils, rmm.pylibrmm.helper, rmm.pylibrmm.memory_resource._memory_resource, rmm.pylibrmm.memory_resource.experimental, rmm.pylibrmm.cuda_stream_pool, rmm.pylibrmm.device_buffer, rmm.librmm._logger, rmm.pylibrmm.logger, nvtx._lib.lib, nvtx._lib.profiler, pylibcudf.libcudf.types, pylibcudf.types, pylibcudf.libcudf.aggregation, pylibcudf.aggregation, pylibcudf.gpumemoryview, pylibcudf.utils, pylibcudf._interop_helpers, pylibcudf.table, pylibcudf.filling, pylibcudf.traits, pylibcudf.column, pylibcudf.scalar, pylibcudf.libcudf.binaryop, pylibcudf.binaryop, pylibcudf.column_factories, pylibcudf.concatenate, pylibcudf.context, pylibcudf.contiguous_split, pylibcudf.libcudf.copying, pylibcudf.copying, pylibcudf.libcudf.datetime, pylibcudf.datetime, pylibcudf.experimental._join_streams, pylibcudf.libcudf.expressions, pylibcudf.expressions, pylibcudf.groupby, pylibcudf.hashing, pylibcudf.interop, pylibcudf.io.datasource, pylibcudf.libcudf.io.json, pylibcudf.libcudf.io.types, pylibcudf.io.types, pylibcudf.io.avro, pylibcudf.io.csv, pylibcudf.io.parquet, pylibcudf.io.text, pylibcudf.libcudf.io.hybrid_scan, pylibcudf.io.experimental.hybrid_scan, pylibcudf.io.json, pylibcudf.io.orc, pylibcudf.io.parquet_metadata, pylibcudf.io.timezone, pylibcudf.libcudf.join, pylibcudf.join, pylibcudf.json, pylibcudf.libcudf.labeling, pylibcudf.labeling, pylibcudf.libcudf.lists.combine, pylibcudf.libcudf.lists.contains, pylibcudf.lists, pylibcudf.merge, pylibcudf.null_mask, pylibcudf.nvtext.byte_pair_encode, pylibcudf.nvtext.deduplicate, pylibcudf.nvtext.edit_distance, pylibcudf.nvtext.generate_ngrams, pylibcudf.nvtext.jaccard, pylibcudf.nvtext.minhash, pylibcudf.nvtext.ngrams_tokenize, pylibcudf.nvtext.normalize, pylibcudf.nvtext.replace, pylibcudf.libcudf.nvtext.stemmer, pylibcudf.nvtext.stemmer, pylibcudf.nvtext.tokenize, pylibcudf.nvtext.wordpiece_tokenize, pylibcudf.libcudf.partitioning, pylibcudf.partitioning, pylibcudf.prefetch, pylibcudf.quantiles, pylibcudf.libcudf.reduce, pylibcudf.reduce, pylibcudf.libcudf.replace, pylibcudf.replace, pylibcudf.reshape, pylibcudf.rolling, pylibcudf.libcudf.round, pylibcudf.round, pylibcudf.search, pylibcudf.sorting, pylibcudf.libcudf.stream_compaction, pylibcudf.stream_compaction, pylibcudf.strings.attributes, pylibcudf.libcudf.strings.char_types, pylibcudf.strings.capitalize, pylibcudf.strings.case, pylibcudf.strings.char_types, pylibcudf.libcudf.strings.combine, pylibcudf.strings.combine, pylibcudf.strings.regex_program, pylibcudf.strings.contains, pylibcudf.strings.convert.convert_booleans, pylibcudf.strings.convert.convert_datetime, pylibcudf.strings.convert.convert_durations, pylibcudf.strings.convert.convert_fixed_point, pylibcudf.strings.convert.convert_floats, pylibcudf.strings.convert.convert_integers, pylibcudf.strings.convert.convert_ipv4, pylibcudf.strings.convert.convert_lists, pylibcudf.strings.convert.convert_urls, pylibcudf.strings.extract, pylibcudf.strings.find, pylibcudf.strings.find_multiple, pylibcudf.strings.findall, pylibcudf.strings.padding, pylibcudf.libcudf.strings.regex_flags, pylibcudf.strings.regex_flags, pylibcudf.strings.repeat, pylibcudf.strings.replace, pylibcudf.strings.replace_re, pylibcudf.strings.reverse, pylibcudf.libcudf.strings.side_type, pylibcudf.strings.side_type, pylibcudf.strings.slice, pylibcudf.strings.split.partition, pylibcudf.strings.split.split, pylibcudf.strings.strip, pylibcudf.libcudf.strings.translate, pylibcudf.strings.translate, pylibcudf.strings.wrap, pylibcudf.transform, pylibcudf.transpose, pylibcudf.libcudf.unary, pylibcudf.unary, pylibcudf.utilities, numba.types.itertools, cudf._lib.strings_udf, pyarrow._feather, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, sklearn.datasets._svmlight_format_fast, sklearn.utils._random, sklearn.utils._vector_sentinel, sklearn.feature_extraction._hashing_fast, pylibraft.common.cuda, pylibraft.common.handle, cuml.internals.logger, cuml.internals.internals, cuml.cluster.agglomerative, cuml.cluster.dbscan, cuml.cluster.hdbscan.hdbscan, cuml.cluster.kmeans, cuml.cluster.spectral_clustering, cuml.datasets.arima, cuml.datasets.regression, cuml.decomposition.pca, cuml.decomposition.tsvd, cuml.internals.treelite, cuml.fil.fil, cuml.metrics.cluster.adjusted_rand_index, cuml.metrics.cluster.completeness_score, cuml.metrics.cluster.entropy, cuml.metrics.cluster.homogeneity_score, cuml.metrics.cluster.mutual_info_score, cuml.metrics.pairwise_distances, cuml.metrics.cluster.silhouette_score, cuml.metrics.cluster.v_measure, cuml.metrics.kl_divergence, cuml.metrics.trustworthiness, cuml.ensemble.randomforest_common, cuml.explainer.base, cupy_backends.cuda.libs.cublas, cuml.solvers.cd, cuml.solvers.qn, cuml.solvers.sgd, cuml.linear_model.linear_regression, cuml.linear_model.ridge, cuml.explainer.kernel_shap, cuml.explainer.permutation_shap, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, cuml.explainer.tree_shap, cuml.manifold.spectral_embedding, cuml.manifold.t_sne, cuml.manifold.umap.umap, cuml.neighbors.nearest_neighbors, cuml.neighbors.kneighbors_classifier, cuml.neighbors.kneighbors_regressor, cuml.svm.linear, cuml.svm.svm_base, cuml.tsa.arima, cuml.tsa.holtwinters, cuml.tsa.stationarity, cuml.tsa.auto_arima, scipy.fftpack.convolve, zstandard.backend_c, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, pyarrow._acero, pyarrow._csv, pyarrow._json, pyarrow._substrait, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, pyarrow._orc, tornado.speedups, msgpack._cmsgpack, raft_dask.common.comms_utils, raft_dask.common.nccl, ucxx._lib.arr, ucxx._lib.libucxx, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.linear_model._sag_fast, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.utils._fast_dict, sklearn.cluster._hierarchical_fast, sklearn.cluster._k_means_common, sklearn.cluster._k_means_elkan, sklearn.cluster._k_means_lloyd, sklearn.cluster._k_means_minibatch, sklearn.cluster._dbscan_inner, sklearn.neighbors._partition_nodes, sklearn.neighbors._ball_tree, sklearn.neighbors._kd_tree, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, sklearn.cluster._hdbscan._tree, sklearn.cluster._hdbscan._linkage, sklearn.cluster._hdbscan._reachability, sklearn._isotonic, sklearn.tree._utils, sklearn.tree._tree, sklearn.tree._partitioner, sklearn.tree._splitter, sklearn.tree._criterion, sklearn.neighbors._quad_tree, sklearn.manifold._barnes_hut_tsne, sklearn.manifold._utils, sklearn.ensemble._gradient_boosting, sklearn.ensemble._hist_gradient_boosting.common, sklearn.ensemble._hist_gradient_boosting._gradient_boosting, sklearn.ensemble._hist_gradient_boosting._binning, sklearn.ensemble._hist_gradient_boosting._bitset, sklearn.ensemble._hist_gradient_boosting.histogram, sklearn.ensemble._hist_gradient_boosting._predictor, sklearn.ensemble._hist_gradient_boosting.splitting, cupy_backends.cuda.libs.curand, cupy_backends.cuda.libs.cusparse, cuml.common.opg_data_utils_mg, cuml.linear_model.base_mg, cuml.linear_model.linear_regression_mg (total: 554)
./scripts/test.sh: line 110: 31831 Segmentation fault      (core dumped) pytest -v --timeout=120 --quick_run packages/cuml/python/cuml/tests/dask

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/cuml/python/cuml/tests/dask/test_dask_base.py and rerun test_get_combined_model for the reported LinearRegression and float32/float64 cases. Inspect the linked nightly workflow output and Dask worker timeout stacks to identify the failure path. Done means the affected nightly cases complete without exceeding the 120-second timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.