docling-project / docling-project/docling-jobkit
Ray orchestrator wedges permanently when _initialize_ray_runtime fails after ray.init() succeeds
- Dominant language
- Python
- Stars
- 36
- Forks
- 47
- Avg merge
- 7d 3h
- Merged PRs (30d)
- 17
Description
## Problem
When `_initialize_ray_runtime` raises after `ray.init()` has succeeded but before `_bind_dispatcher()` completes (e.g. transient Ray client data-channel hiccup during `serve.start` or `deploy_processor`), the half-initialized Ray client stays registered in the process. The dispatcher supervisor's retry loop introduced in #122 then calls `ray.init()` again on every iteration and fails permanently with:
```
The client has already connected to the cluster with allow_multiple=True. Please set allow_multiple=True to proceed
```
The supervisor never recovers without a process restart.
## Repro
1. Run the Ray orchestrator against a remote cluster (`ray://...:10001`).
2. After `ray.init()` succeeds, inject a failure between `ray.init` and `_bind_dispatcher` — e.g. drop the Ray client data channel mid-init, or transiently lose the head node, so `serve.start` / `deploy_processor` raises.
3. The `except BaseException` block at `docling_jobkit/orchestrators/ray/orchestrator.py:313-318` clears `self.dispatcher` and `self.deployment_handle` but **does not** call `ray.shutdown()`.
4. The supervisor's next iteration calls `_initialize_ray_runtime` again. `ray.is_initialized()` returns False, so it calls `ray.init()` — which fails with the `allow_multiple=True` error above. Every subsequent retry fails identically.
## Impact
Observed on a KubeRay deployment with a single docling-serve HTTP pod: a Ray client data-channel reconnect wedged the orchestrator for hours until the pod was restarted manually. `/health` returned 200 OK throughout because FastAPI was alive; every `/v1/convert/source/async` returned 503.
## Proposed fix
Best-effort `ray.shutdown()` in the `except BaseException` block of `_initialize_ray_runtime` so the next supervisor iteration starts from a clean slate. Wrapped in `try/except Exception` so it cannot mask the original initialization error.
Happy to send a PR with a regression test if this analysis matches the maintainers' read.
Contributor guide
Research direction
Read docling_jobkit/orchestrators/ray/orchestrator.py around _initialize_ray_runtime and its exception block at lines 313-318. Trace the supervisor retry path introduced in #122, then add a regression test for a failure after ray.init() succeeds. Done means the initialization failure is preserved, cleanup is best-effort, and the next retry can initialize cleanly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100