Dedicated LocalBackend vLLM server can become unreachable during LoRA adapter reload

Summary

In LocalBackend dedicated mode with PipelineTrainer, some runs fail or become unhealthy immediately after an early training step when ART tries to reload the newly saved LoRA adapter into the dedicated vLLM server.

The observed failure is an httpx.ConnectError: All connection attempts failed from UnslothService._reload_adapter(...). After this starts happening, rollout workers also repeatedly fail with generic Connection error while calling /chat/completions.

This was seen after upgrading ART to main commit 48b2e5f6c384a62b44f34e1472e5fb1eeaa3474a.

Setup

• ART commit: 48b2e5f6c384a62b44f34e1472e5fb1eeaa3474a
• Backend: LocalBackend
• Trainer: PipelineTrainer
• Mode: dedicated trainer/inference GPUs
• Hardware: 1 node, 2x H200; one GPU for trainer and one GPU for inference
• Base model class: Llama 3.1 8B Instruct
• Training: GRPO/CISPO via Unsloth
• vLLM from the ART lock at this commit (vllm 0.17.0+art1 in the generated lock)
• Checkpoint saving enabled
• eval_at_start enabled

No private data is needed to reproduce the failure mode; this appears to be an infrastructure / adapter reload stability issue.

Symptoms

One run failed after the first training step with this stack:

Pipeline stage failed: All connection attempts failed
ExceptionGroup: unhandled errors in a TaskGroup
  File ".../art/pipeline_trainer/trainer.py", line 467, in _training_stage
    result = await self.backend.train(...)
  File ".../art/unsloth/service.py", line 628, in _train_dedicated
    await self._reload_adapter(checkpoint_dir, new_step)
  File ".../art/unsloth/service.py", line 455, in _reload_adapter
    response = await client.post(...)
  File ".../httpx/_client.py", line 1859, in post
    return await self.request(...)
httpx.ConnectError: All connection attempts failed

Other concurrent runs on the same setup remained marked as running, but stopped making useful training progress and emitted repeated rollout failures like:

Worker N: rollout failed: Connection error.
Retrying request to /chat/completions ...

Expected behavior

If the dedicated vLLM server is temporarily unavailable during adapter reload, ART should either:

1. Wait for the server to become healthy again before resuming rollouts/training, or
2. Restart/recover the dedicated inference server, or
3. Fail with a clearer diagnostic that includes vLLM process health and recent server logs.

A transient adapter reload/server restart should not leave the pipeline in a state where rollout workers keep generating connection errors indefinitely.

Actual behavior

At least one run hard-failed during _reload_adapter(...) after saving a checkpoint. Other runs kept running at the Sky/job level but repeatedly failed rollout calls to the local vLLM endpoint and did not advance training meaningfully.

Possibly related

This was found while investigating forked LocalBackend/PipelineTrainer runs. There is already an open fork-specific issue here:

• #651

The adapter reload/connectivity failure described in this issue is separate from whether the run started from the intended fork checkpoint. Even after ignoring the fork correctness problem, the dedicated vLLM server becoming unreachable during or after adapter reload appears to be a separate stability issue.

Suggested debugging additions

It would help if _reload_adapter(...) logged or surfaced:

• Target URL/endpoint being called
• Whether the vLLM subprocess is still alive
• Last lines of the dedicated vLLM log file
• Health check result before and after adapter reload
• The checkpoint path and step being reloaded

Start in art/unsloth/service.py at _reload_adapter(...) and follow its call from _train_dedicated; inspect the dedicated server lifecycle and the _training_stage path in art/pipeline_trainer/trainer.py. Reproduce the adapter reload failure with the listed LocalBackend and PipelineTrainer setup, then verify that a temporarily unavailable vLLM server recovers or produces a diagnostic containing its health and recent logs.