Azure / Azure/unbounded

[orca] Wire cachestore HealthCheck into /readyz as a periodic readiness probe

Open
#202 0 comments 0 reactions 0 assignees View on GitHub
enhancement go
Dominant language
Go
Stars
28
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
55

Description

### Background

Orca's `/readyz` gates readiness on `cachestoreReady`, a **frozen boot bool** set once after the cachestore `SelfTest` passes (`internal/orca/app/app.go:208`, read by `isReady()` at `app.go:400-402`). It is never re-evaluated. An instance that loses connectivity to the cachestore *after* boot stays `Ready` indefinitely and keeps receiving edge traffic it cannot serve or fill.

The boot-time `SelfTest` (`internal/orca/cachestore/s3/s3.go:171`) is retained as a startup write/read/delete preflight (fail-fast on bad creds / non-writable bucket). This issue is about adding a *continuous* signal on top of it.

### Proposal

Add a single cheap probe to the `CacheStore` interface and use it in both places:

```go
// HealthCheck does a cheap reachability/credentials probe of the
// backend (e.g. HeadBucket / GetBucketVersioning). Used at boot and
// periodically for readiness.
HealthCheck(ctx context.Context) error
```

- A background prober goroutine (mirror `cluster.refreshLoop`, `internal/orca/cluster/cluster.go:421`): ticker at a configurable interval, store an `atomic.Bool` healthy flag plus last-error/last-success for observability, with consecutive-failure hysteresis (cf. `maxStalePeerRefreshes`) so a single transient blip does not flap Service endpoints.
- `isReady()` becomes `cachestoreHealthy.Load() && Cluster.HasInitialSnapshot()`.
- Lifecycle: start after boot; stop in `App.Shutdown` (cancel + wait), like `Cluster.Close` (`app.go:502`).

### Readiness, not liveness (important)

Wire the probe into **`/readyz` only**. Keep `/healthz` a pure process-alive 200. Cachestore connectivity loss must **not** fail liveness: a restart cannot fix an external-dependency outage and would crashloop every replica simultaneously during a cachestore blip. Readiness failure correctly pulls the pod from Service endpoints and auto-rejoins on recovery. The deployment already polls `/readyz` every 5s (`deploy/orca/04-deployment.yaml.tmpl:87`), so no manifest change is required (a dedicated `cachestore_health` interval/threshold config knob can be added).

### Scope

- Add `HealthCheck(ctx)` to `cachestore.CacheStore`; implement in `cachestore/s3` (read-only `HeadBucket`/`GetBucketVersioning`).
- Have boot call `HealthCheck` (keep `SelfTest` as the write/read/delete preflight).
- Add the prober + atomic flag + hysteresis in `internal/orca/app`; rewire `isReady()`.
- Config knobs (interval, timeout, failure threshold) under cachestore config, mirroring `cluster` style.
- Update the in-memory/test stubs (`internal/orca/fetch/fetch_test.go:229`, the `WithSkipCachestoreSelfTest` test path).
- Tests: unit (flag flips after N failures, recovers on success; `isReady` reflects it) + integration (pause the Garage testcontainer, assert `/readyz` -> 503, restore, assert recovery).

### Out of scope

- Liveness behavior (`/healthz` stays unconditional).
- Boot hard-fail behavior of `SelfTest` (unchanged).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.