alibaba / alibaba/ROCK

[Feature] 能够在客户端知晓拉取镜像超时或失败

Open
#256 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
485
Forks
81
Avg merge
16h 12m
Merged PRs (30d)
8

Description

**Feature Category**
- [ ] Sandbox
- [ ] Actions
- [ ] Deployments
- [x] SDK & API
- [ ] Envhub
- [ ] CLI
- [ ] Performance & Optimization
- [ ] Documentation & Examples

**Problem Statement**

在Admin处已经看到了沙箱拉取失败的报错(下面贴有),但是客户端对此一无所知,依旧在get_status尝试获取is_alive状态。

**Proposed Solution**

1)能够透出这种错误给客户端知晓;
2)README或者其他文档中能够提及这个问题并提供手工验证方法;

**Detailed Feature Description**

2026-01-15 11:32:15,984 ERROR worker.py:422 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::SandboxActor.start() (pid=18618, ip=127.0.0.1, actor_id=6267a28e00f115e344e16abd01000000, repr=)
File "/tmp/ray/session_2026-01-15_11-28-33_624678_18263/runtime_resources/working_dir_files/_ray_pkg_2bb51da3ce7b67cc/rock/utils/docker.py", line 37, in pull_image
raise subprocess.CalledProcessError(e.returncode, e.cmd, e.output, e.stderr) from None
subprocess.CalledProcessError: Command '['docker', 'pull', 'python:3.11']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

ray::SandboxActor.start() (pid=18618, ip=127.0.0.1, actor_id=6267a28e00f115e344e16abd01000000, repr=)
File "/Users/timandes/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/Users/timandes/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/timandes/Projects/ROCK/ROCK/rock/sandbox/sandbox_actor.py", line 129, in start
raise ex
File "/Users/timandes/Projects/ROCK/ROCK/rock/sandbox/sandbox_actor.py", line 126, in start
await self._deployment.start()
File "/tmp/ray/session_2026-01-15_11-28-33_624678_18263/runtime_resources/working_dir_files/_ray_pkg_2bb51da3ce7b67cc/rock/deployments/docker.py", line 280, in start
await loop.run_in_executor(executor, self._pull_image)
File "/Users/timandes/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/ray/session_2026-01-15_11-28-33_624678_18263/runtime_resources/working_dir_files/_ray_pkg_2bb51da3ce7b67cc/rock/deployments/docker.py", line 183, in _pull_image
raise DockerPullError(msg) from e
rock.rocklet.exceptions.DockerPullError: Failed to pull image python:3.11. Error: bash: line 1: cd: /tmp/ray/session_2026-01-15_11-28-33_624678_18263/runtime_resources/working_dir_files/_ray_pkg_2bb51da3ce7b67cc: No such file or directory
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Output:
(raylet) [2026-01-15 11:32:15,975 E 18284 9538428] (raylet) file_system_monitor.cc:116: /tmp/ray/session_2026-01-15_11-28-33_624678_18263 is over 95% full, available space: 2.00325 GB; capacity: 460.432 GB. Object creation will fail if spilling is required.
(SandboxActor pid=18618) [python:3.11] Image pull took 15.122 s
(SandboxActor pid=18618) 2026-01-15 11:32:15,974 ERROR:sandbox_actor.py:128 [rock.sandbox.sandbox_actor] [] [] -- start deployment failed: Failed to pull image python:3.11. Error: bash: line 1: cd: /tmp/ray/session_2026-01-15_11-28-33_624678_18263/runtime_resources/working_dir_files/_ray_pkg_2bb51da3ce7b67cc: No such file or directory
(SandboxActor pid=18618) Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with rock/sandbox/sandbox_actor.py and rock/deployments/docker.py, especially the deployment start and image-pull error paths shown in the report. Trace how get_status observes sandbox liveness and review the README or related documentation. Done means clients can learn when an image pull fails or times out, with a documented manual verification method.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
api, backend, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.