THUDM / THUDM/slime

[Bug] Actor unavailable error

Open
#1,739 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
8.5k
Forks
1.3k
Avg merge
5h 36m
Merged PRs (30d)
22

Description

Bug Description

Traceback (most recent call last):
2026-03-18 05:25:02

2855e530…vdumckda
File "/root/slime_siqi/train.py", line 100, in
2026-03-18 05:25:02

2855e530…vdumckda
train(args)
2026-03-18 05:25:02

2855e530…vdumckda
File "/root/slime_siqi/train.py", line 69, in train
2026-03-18 05:25:02

2855e530…vdumckda
rollout_data_ref = ray.get(rollout_manager.generate.remote(rollout_id))
2026-03-18 05:25:02

2855e530…vdumckda
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-03-18 05:25:02

2855e530…vdumckda
File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
2026-03-18 05:25:02

2855e530…vdumckda
return fn(*args, **kwargs)
2026-03-18 05:25:02

2855e530…vdumckda
^^^^^^^^^^^^^^^^^^^
2026-03-18 05:25:03

2855e530…vdumckda
File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
2026-03-18 05:25:03

2855e530…vdumckda
return func(*args, **kwargs)
2026-03-18 05:25:03

2855e530…vdumckda
^^^^^^^^^^^^^^^^^^^^^
2026-03-18 05:25:03

2855e530…vdumckda
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2981, in get
2026-03-18 05:25:03

2855e530…vdumckda
values, debugger_breakpoint = worker.get_objects(
2026-03-18 05:25:03

2855e530…vdumckda
^^^^^^^^^^^^^^^^^^^
2026-03-18 05:25:03

2855e530…vdumckda
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 1014, in get_objects
2026-03-18 05:25:03

2855e530…vdumckda
raise value
2026-03-18 05:25:03

2855e530…vdumckda
ray.exceptions.ActorUnavailableError: The actor ed2ff6eb7e211cc420c657b002000000 is unavailable: The actor is temporarily unavailable: RpcError: RPC error: Socket closed rpc_code: 14. The task may or may not have been executed on the actor.

Steps to Reproduce

I ran qwen3-1.7b training on math dataset, in step 9, this error occurrs

Expected Behavior

The program shouldn't collapse

Actual Behavior

the program exit

Environment
  • slime version:
  • Python version:
  • PyTorch version:
  • CUDA/ROCm version:
  • GPU type and count:
  • OS:
  • SGLang version (if relevant):
  • Megatron-LM version (if relevant):
Logs

Additional Context

No response

Pre-submission Checklist
  • I have read the CONTRIBUTING.md and understand the collaboration scope.
  • I have read the documentation and my issue is not addressed there.
  • I have searched for existing issues and this is not a duplicate.
  • I have provided a minimal, reproducible example.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with train.py around lines 69 and 100, especially the ray.get call for rollout_manager.generate, and reproduce the qwen3-1.7b math training run through step 9. Trace why the actor becomes unavailable and document a verified outcome where the program no longer exits unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.