vllm-project / vllm-project/aibrix

RayClusterFleet creates a new RayCluster before deleting the old one after a Pod failure

Open
#2,435 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🐛 Describe the bug

For a running `RayClusterFleet`, when one Pod inside the managed `RayCluster` is killed (e.g. force-delete a worker Pod), AIBrix treats the RayCluster as unrecoverable and should recreate it.

**Expected recreate semantics:** delete/tear down the unhealthy old `RayCluster` first, then create a new `RayCluster`.

**Actual behavior:** AIBrix creates a **new** `RayCluster` immediately while the **old** `RayCluster` (and remaining Pods) are still present. The old cluster is cleaned up later (or may linger), so old and new RayClusters coexist for a period of time.

This can cause temporary double resource usage (e.g. GPUs) and is inconsistent with a delete-then-create recovery path.

**Related code / analysis**

1. `RayClusterReplicaSet` marks a previously provisioned but no-longer-ready RayCluster as inactive (`isClusterActive` / `filterActiveClusters` in `pkg/controller/rayclusterreplicaset/rayclusterreplicaset_utils.go`), then scales up a new RayCluster because active replicas < desired.
2. The inactive old RayCluster is not necessarily deleted as part of that scale-up path, so a new RayCluster can appear first.
3. Separately, for `strategy.type: Recreate`, `oldPodsRunning` in `pkg/controller/rayclusterfleet/recreate.go` also appears incorrect vs upstream Kubernetes Deployment Recreate (it only waits when a cluster is deleting **and** still Ready), which can similarly allow create-before-delete during rollouts. The primary scenario reported here is **Pod failure recovery** on a running Fleet, not necessarily a template rollout.

### Steps to Reproduce

1. Deploy AIBrix `0.6.0` with RayClusterFleet / distributed-inference controllers enabled, and KubeRay installed.
2. Create a `RayClusterFleet` with `spec.replicas: 1` and a RayCluster template that has multiple Pods (e.g. 1 head + N workers). Wait until the Fleet is Available and the RayCluster is Ready.
3. Force-delete one Pod belonging to that RayCluster, e.g.:

```bash
kubectl delete pod --force --grace-period=0
```

4. Watch RayClusters and Pods:

```bash
kubectl get raycluster -w
kubectl get rayclusterreplicaset -w
kubectl get pods -l ray.io/is-ray-node=yes -w
```

5. Observe that a **new** RayCluster is created while the **old** RayCluster is still present.

### Expected behavior

After a Pod in the RayCluster is killed and the RayCluster becomes unhealthy:

1. AIBrix should delete / tear down the old unhealthy `RayCluster` first.
2. Only after the old RayCluster is gone (or fully terminating in a controlled delete-first sequence), create a new `RayCluster` to restore desired replicas.
3. There should be no period where a healthy-path new RayCluster is started while the old RayCluster is still fully present and consuming resources.

### Actual behavior

1. After killing one Pod, the old RayCluster becomes not Ready / inactive from AIBrix's perspective.
2. AIBrix immediately creates a **new** RayCluster to satisfy desired replicas.
3. The old RayCluster is still present at that time (delete happens later or is incomplete from the user's perspective).
4. Result: create-new-first, rather than delete-old-then-create-new.

### Environment

```text
- AIBrix version: 0.6.0
- Deployment environment: Kubernetes 1.34.1
- Cloud provider (if applicable): N/A / private cluster
- KubeRay version: v1.2.1-patch-20250726
- Workload: RayClusterFleet managing 1 RayCluster (multi-Pod head/worker)
```

### Additional context

- User expectation for failure recovery: kill one Pod → AIBrix kills the whole RayCluster → then starts a new RayCluster.
- Observed: kill one Pod → AIBrix starts a new RayCluster directly (old one still around).
- Suggested fix direction:
- On RayCluster becoming inactive/unhealthy, explicitly delete the old RayCluster (or scale down) **before** creating a replacement.
- Ensure recreate / recovery paths never count an inactive-but-still-existing RayCluster as “gone” for the purpose of allowing a new create.
- Add a test that force-deletes a worker/head Pod and asserts no second RayCluster exists until the first is deleted.

Contributor guide

Open the contributing guide

Research direction

Start with pkg/controller/rayclusterreplicaset/rayclusterreplicaset_utils.go, especially isClusterActive and filterActiveClusters, then inspect pkg/controller/rayclusterfleet/recreate.go and its oldPodsRunning logic. Reproduce the worker or head Pod deletion with the kubectl commands in the issue and trace when the old RayCluster is considered inactive versus deleted. Done means recovery deletes or tears down the old RayCluster before creating a replacement, with a regression test asserting no second RayCluster exists first.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.