kvcache-ai / kvcache-ai/AgentENV
Add scheduler HA using Kubernetes Lease leader election
- Dominant language
- Rust
- Stars
- 3.5k
- Forks
- 309
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 66
Description
### Affected component
Gateway / scheduler
### Problem statement
AgentENV currently requires one read/write scheduler. Redis-backed query-only replicas preserve existing sandbox routing, but scheduling, node inspection, assignment writes, and P2P APIs remain unavailable if the primary fails.
Running multiple read/write schedulers is unsafe because heartbeat state is replica-local. This causes resource limits to be applied inconsistently, as reported in #191.
### Use case
Production Kubernetes deployments need scheduler redundancy for runtime failures, upgrades, and node maintenance without operating multiple active schedulers or introducing a custom replication protocol.
### Current behavior and workarounds
The Kubernetes deployment runs one scheduler replica. Operators can add query-only replicas backed by Redis, but these provide data-plane lookup availability only. The practical workaround is restarting the single primary scheduler and waiting for heartbeats to rebuild its state.
### Desired behavior
- Multiple scheduler pods may be deployed.
- Exactly one scheduler performs scheduling and processes node reports.
- Standbys participate only in leader election and health reporting.
- A standby automatically takes leadership after leader failure.
- The promoted leader rebuilds mappings and observations from node heartbeats.
- Missing observations during recovery are not treated as unlimited capacity.
- Leadership loss immediately stops the old leader from serving.
- Liveness remains healthy on standbys; readiness exposes only the leader.
- Failover completes within the configured lease duration plus heartbeat recovery time.
- Three scheduler replicas pass the resource-limit scenario from #191.
### Proposed approach
Use `client-go` leader election with a `coordination.k8s.io/Lease`.
Add scheduler configuration for:
```json
{
"leader_election": {
"enabled": true,
"lease_name": "agentenv-scheduler",
"lease_namespace": "agentenv-system",
"lease_duration": "15s",
"renew_deadline": "10s",
"retry_period": "2s"
}
}
```
On leadership acquisition:
- Start discovery and the scheduler service.
- Accept heartbeats immediately.
- Schedule only onto nodes with fresh observations.
- Return `Unavailable` rather than false `NotFound` results while in-memory bindings are rebuilding.
On leadership loss:
- Mark the scheduler service non-serving.
- Stop accepting RPCs and exit so existing gRPC connections reconnect.
Update Kubernetes manifests to use three replicas, grant Lease RBAC permissions, and use separate liveness and leader-readiness checks. Leader election and `--query-only` should be mutually exclusive.
### Compatibility and operational impact
API/config changes: Add optional scheduler leader-election configuration. No protobuf API change is expected.
Snapshot or storage format changes: None.
New host/runtime requirements: Kubernetes Lease API access and corresponding RBAC permissions.
Upgrade and rollback considerations: Leader election should remain disabled by default. Existing single-replica and query-only deployments continue to work. Rollback requires scaling the scheduler back to one replica.
### Alternatives considered
- Active-active schedulers with all observations stored in Redis: larger consistency and operational scope.
- Chain replication between schedulers: requires custom membership, failure detection, reconfiguration, and fencing.
- Heartbeat fanout to every scheduler: duplicates state and still permits concurrent scheduling races.
- Current primary plus query-only replicas: does not provide control-plane failover.
### Pre-submission checklist
- [x] I searched existing issues and discussions and did not find a duplicate. #191 is related but covers the existing correctness bug.
- [x] I described a concrete problem and use case, not only a proposed technology.
- [x] I understand that acceptance of the problem does not imply approval of a particular implementation.
Contributor guide
Research direction
Start by tracing the scheduler service, scheduler configuration, Kubernetes manifests, and liveness/readiness checks; inspect the resource-limit scenario from #191. Confirm the existing heartbeat and query-only behavior before designing the Lease integration. Done means three replicas elect one serving leader, failover rebuilds safe state, standbys remain live but unready, and the resource-limit scenario passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, kubernetes, rust
- Domain
- cloud, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100