docling-project / docling-project/docling-jobkit

Ray orchestrator wedges permanently when _initialize_ray_runtime fails after ray.init() succeeds

Open Beginner friendly
#160 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36
Forks
47
Avg merge
7d 3h
Merged PRs (30d)
17

Description

## Problem

When `_initialize_ray_runtime` raises after `ray.init()` has succeeded but before `_bind_dispatcher()` completes (e.g. transient Ray client data-channel hiccup during `serve.start` or `deploy_processor`), the half-initialized Ray client stays registered in the process. The dispatcher supervisor's retry loop introduced in #122 then calls `ray.init()` again on every iteration and fails permanently with:

```
The client has already connected to the cluster with allow_multiple=True. Please set allow_multiple=True to proceed
```

The supervisor never recovers without a process restart.

## Repro

1. Run the Ray orchestrator against a remote cluster (`ray://...:10001`).
2. After `ray.init()` succeeds, inject a failure between `ray.init` and `_bind_dispatcher` — e.g. drop the Ray client data channel mid-init, or transiently lose the head node, so `serve.start` / `deploy_processor` raises.
3. The `except BaseException` block at `docling_jobkit/orchestrators/ray/orchestrator.py:313-318` clears `self.dispatcher` and `self.deployment_handle` but **does not** call `ray.shutdown()`.
4. The supervisor's next iteration calls `_initialize_ray_runtime` again. `ray.is_initialized()` returns False, so it calls `ray.init()` — which fails with the `allow_multiple=True` error above. Every subsequent retry fails identically.

## Impact

Observed on a KubeRay deployment with a single docling-serve HTTP pod: a Ray client data-channel reconnect wedged the orchestrator for hours until the pod was restarted manually. `/health` returned 200 OK throughout because FastAPI was alive; every `/v1/convert/source/async` returned 503.

## Proposed fix

Best-effort `ray.shutdown()` in the `except BaseException` block of `_initialize_ray_runtime` so the next supervisor iteration starts from a clean slate. Wrapped in `try/except Exception` so it cannot mask the original initialization error.

Happy to send a PR with a regression test if this analysis matches the maintainers' read.

Contributor guide

Open the contributing guide

Research direction

Read docling_jobkit/orchestrators/ray/orchestrator.py around _initialize_ray_runtime and its exception block at lines 313-318. Trace the supervisor retry path introduced in #122, then add a regression test for a failure after ray.init() succeeds. Done means the initialization failure is preserved, cleanup is best-effort, and the next retry can initialize cleanly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.