alibaba / alibaba/ROLL

previous Ray instance has since crashed

Open
#147 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
312
Avg merge
1h 2m
Merged PRs (30d)
2

Description

2025-08-20 02:28:57,027 INFO worker.py:1853 -- Failed to connect to the default Ray cluster address at 10.164.31.145:6379. This is most likely due to a previous Ray instance that has since crashed. To reset the default address to connect to, run `ray stop` or restart Ray with `ray start`.
Traceback (most recent call last):
File "/usr/local/conda/lib/python3.10/site-packages/ray/_private/worker.py", line 1844, in init
_global_node = ray._private.node.Node(
File "/usr/local/conda/lib/python3.10/site-packages/ray/_private/node.py", line 287, in __init__
node_info = ray._private.services.get_node_to_connect_for_driver(
File "/usr/local/conda/lib/python3.10/site-packages/ray/_private/services.py", line 482, in get_node_to_connect_for_driver
return global_state.get_node_to_connect_for_driver(node_ip_address)
File "/usr/local/conda/lib/python3.10/site-packages/ray/_private/state.py", line 827, in get_node_to_connect_for_driver
return self.global_state_accessor.get_node_to_connect_for_driver(
File "python/ray/includes/global_state_accessor.pxi", line 279, in ray._raylet.GlobalStateAccessor.get_node_to_connect_for_driver
RuntimeError: b'RPC Error message: Deadline Exceeded; RPC Error details: '

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workdir/examples/start_agentic_pipeline.py", line 35, in
main()
File "/workdir/examples/start_agentic_pipeline.py", line 26, in main
init()
File "/workdir/roll/distributed/scheduler/initialize.py", line 66, in init
ray.init(
File "/usr/local/conda/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/usr/local/conda/lib/python3.10/site-packages/ray/_private/worker.py", line 1860, in init
raise ConnectionError
ConnectionError

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with examples/start_agentic_pipeline.py at line 35 and roll/distributed/scheduler/initialize.py at line 66, where ray.init is called. Reproduce the failure and inspect the Ray connection and RPC timeout in the traceback, using the reported ray stop and ray start commands as initial context. Done means the cause is identified and the pipeline can initialize without the ConnectionError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.