kvcache-ai / kvcache-ai/AgentENV

Add scheduler HA using Kubernetes Lease leader election

Open
#259 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.5k
Forks
309
Avg merge
1d 9h
Merged PRs (30d)
66

Description

### Affected component

Gateway / scheduler

### Problem statement

AgentENV currently requires one read/write scheduler. Redis-backed query-only replicas preserve existing sandbox routing, but scheduling, node inspection, assignment writes, and P2P APIs remain unavailable if the primary fails.

Running multiple read/write schedulers is unsafe because heartbeat state is replica-local. This causes resource limits to be applied inconsistently, as reported in #191.

### Use case

Production Kubernetes deployments need scheduler redundancy for runtime failures, upgrades, and node maintenance without operating multiple active schedulers or introducing a custom replication protocol.

### Current behavior and workarounds

The Kubernetes deployment runs one scheduler replica. Operators can add query-only replicas backed by Redis, but these provide data-plane lookup availability only. The practical workaround is restarting the single primary scheduler and waiting for heartbeats to rebuild its state.

### Desired behavior

- Multiple scheduler pods may be deployed.
- Exactly one scheduler performs scheduling and processes node reports.
- Standbys participate only in leader election and health reporting.
- A standby automatically takes leadership after leader failure.
- The promoted leader rebuilds mappings and observations from node heartbeats.
- Missing observations during recovery are not treated as unlimited capacity.
- Leadership loss immediately stops the old leader from serving.
- Liveness remains healthy on standbys; readiness exposes only the leader.
- Failover completes within the configured lease duration plus heartbeat recovery time.
- Three scheduler replicas pass the resource-limit scenario from #191.

### Proposed approach

Use `client-go` leader election with a `coordination.k8s.io/Lease`.

Add scheduler configuration for:

```json
{
"leader_election": {
"enabled": true,
"lease_name": "agentenv-scheduler",
"lease_namespace": "agentenv-system",
"lease_duration": "15s",
"renew_deadline": "10s",
"retry_period": "2s"
}
}
```

On leadership acquisition:

- Start discovery and the scheduler service.
- Accept heartbeats immediately.
- Schedule only onto nodes with fresh observations.
- Return `Unavailable` rather than false `NotFound` results while in-memory bindings are rebuilding.

On leadership loss:

- Mark the scheduler service non-serving.
- Stop accepting RPCs and exit so existing gRPC connections reconnect.

Update Kubernetes manifests to use three replicas, grant Lease RBAC permissions, and use separate liveness and leader-readiness checks. Leader election and `--query-only` should be mutually exclusive.

### Compatibility and operational impact

API/config changes: Add optional scheduler leader-election configuration. No protobuf API change is expected.

Snapshot or storage format changes: None.

New host/runtime requirements: Kubernetes Lease API access and corresponding RBAC permissions.

Upgrade and rollback considerations: Leader election should remain disabled by default. Existing single-replica and query-only deployments continue to work. Rollback requires scaling the scheduler back to one replica.

### Alternatives considered

- Active-active schedulers with all observations stored in Redis: larger consistency and operational scope.
- Chain replication between schedulers: requires custom membership, failure detection, reconfiguration, and fencing.
- Heartbeat fanout to every scheduler: duplicates state and still permits concurrent scheduling races.
- Current primary plus query-only replicas: does not provide control-plane failover.

### Pre-submission checklist

- [x] I searched existing issues and discussions and did not find a duplicate. #191 is related but covers the existing correctness bug.
- [x] I described a concrete problem and use case, not only a proposed technology.
- [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the scheduler service, scheduler configuration, Kubernetes manifests, and liveness/readiness checks; inspect the resource-limit scenario from #191. Confirm the existing heartbeat and query-only behavior before designing the Lease integration. Done means three replicas elect one serving leader, failover rebuilds safe state, standbys remain live but unready, and the resource-limit scenario passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, kubernetes, rust
Domain
cloud, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.