Asynchronous rollout timeout when using server (LLM-as-a-Judge)

I'm using LLM-as-a-Judge to generate the reward, where LLM is running as a Llama 3.1 server, but after rolling out a few samples, I get the following error:

  File "/data/zxz/src/slime/train.py", line 106, in <module>
    train(args)
  File "/data/zxz/src/slime/train.py", line 68, in train
    rollout_data_ref = ray.get(rollout_manager.generate.remote(rollout_id))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/secure/anaconda3/envs/slime/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/secure/anaconda3/envs/slime/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/secure/anaconda3/envs/slime/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/home/secure/anaconda3/envs/slime/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TimeoutError): ray::RolloutManager.generate() (pid=2560774, ip=10.1.3.122, actor_id=cf76d3fadf3d5d0b4b72bc9202000000, repr=<slime.ray.rollout.RolloutManager object at 0x7f73aab64890>)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/zxz/src/slime/slime/ray/rollout.py", line 102, in generate
    data, metrics = self._get_rollout_data(rollout_id=rollout_id)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/zxz/src/slime/slime/ray/rollout.py", line 158, in _get_rollout_data
    data = call_rollout_fn(self.generate_rollout, self.args, rollout_id, self.data_source, evaluation=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/zxz/src/slime/slime/rollout/base_types.py", line 20, in call_rollout_fn
    output = fn(*args, **kwargs, evaluation=evaluation)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/zxz/src/slime/slime/rollout/sglang_rollout.py", line 598, in generate_rollout
    output, aborted_samples = generate_abortable_samples(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/zxz/src/slime/slime/rollout/sglang_rollout.py", line 614, in generate_abortable_samples
    return run(generate_rollout_async(args, rollout_id, data_source))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/zxz/src/slime/slime/utils/async_utils.py", line 36, in run
    return get_async_loop().run(coro)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/zxz/src/slime/slime/utils/async_utils.py", line 20, in run
    return asyncio.run_coroutine_threadsafe(coro, self.loop).result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/secure/anaconda3/envs/slime/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/secure/anaconda3/envs/slime/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
TimeoutError
(SGLangEngine pid=2561938) [2026-01-08 17:07:46] INFO:     10.1.3.122:58324 - "GET /health HTTP/1.1" 200 OK

---------------------------------------
Job 'raysubmit_P52uBMrKVEMCKbBr' failed
---------------------------------------

Status message: Job entrypoint command failed with exit code 1, last available logs (truncated to 20,000 chars):
  File "/data/zxz/src/slime/slime/utils/async_utils.py", line 20, in run
    return asyncio.run_coroutine_threadsafe(coro, self.loop).result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/secure/anaconda3/envs/slime/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/secure/anaconda3/envs/slime/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
TimeoutError
(SGLangEngine pid=2561938) [2026-01-08 17:07:46] INFO:     10.1.3.122:58324 - "GET /health HTTP/1.1" 200 OK

Reproduce the LLM-as-a-Judge rollout with the Llama 3.1 server, then trace RolloutManager.generate and _get_rollout_data in slime/ray/rollout.py through generate_rollout and generate_abortable_samples in slime/ray/rollout/sglang_rollout.py. Check the async execution in slime/utils/async_utils.py and the SGLang /health log; done means the rollout completes without the reported TimeoutError.