gpustack / gpustack/gpustack-operator
todo: the measurement chose the service address so the lease path needs placing
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4
- Forks
- 7
- Avg merge
- 3h 9m
- Merged PRs (30d)
- 213
Description
The measurement is in, and it chose the Service address
Both paths were run against the same failover on one cluster, deleting the Pod labelled
mooncake.io/store-role=leader and recording how long a member was unable to reach a master.
| path | convergence | first error after the delete |
|---|---|---|
A - the member reads the Lease (k8s://<ns>/<lease>), what is rendered today |
60.715 s | 31.411 s |
| B - the member keeps the leader Service address | 60.586 s | 31.410 s |
Path B is 0.13 s apart from Path A, which is inside the noise of a single run. Path B also
works, which was the half that had never been observed: the client's ping loop reconnects to the
same DNS name on its own, and the Service's endpoints have already moved.
So the cost Path A imposes buys nothing. That cost is not small: a member's client has to be built
with STORE_USE_K8S_LEASE, which is why a member group on a vendor image cannot take part in an
HA backend at all under Path A. Path B has no such requirement.
TWO LIMITS ON THAT READING, AND THE FIRST ONE BOUNDS THE CONCLUSION
1. The comparison is dominated by a term neither path controls. Both paths reach their first
error at 31.41 s and converge at ~60.6 s, which means the time is spent on the new leader acquiring
the Lease and beginning to serve - not on either discovery mechanism. The endpoint propagation delay
Path B has to pay (up to ~15 s, from periodSeconds: 5 and failureThreshold: 3) is hidden inside
that window rather than absent from it.
⇒ This conclusion holds while leader election is the slow term. Shorten the Lease duration or
otherwise speed up promotion, and Path B's propagation delay stops being hidden. Anyone tuning
election timing should re-run this comparison rather than inherit its verdict.
2. Path B's mechanism was not directly observed - only its outcome. The e2e case reports
endpoints: no change observed after delete, and that line is BLIND here: its criterion compares
every post-t0 sample against the FIRST post-t0 sample, so an endpoint that switched before the
first sample was taken necessarily reads as "no change". The endpoints moving is still an inference
from upstream source and this operator's probe configuration; what is measured is that a member
recovers.
What is left to do
The decision is made - render the leader Service address. What remains is the code and the
placement of the path that is no longer needed:
- Change what the member receives from
k8s://<ns>/<lease>back to the leader Service address,
which is what the non-HA path already renders. - Decide what happens to the lease-reading path, which is built and works:
- remove it, so there is one way a member finds the leader and no dead branch to maintain; or
- keep it behind something an administrator can select, on the argument that a cluster whose
election is tuned fast enough would prefer it - in which case limit 1 above is the criterion
that selects it, and it belongs in the field's documentation.
The second bullet is a design choice rather than a consequence of the measurement, which is why
this issue stays open after the measurement answered the question it was filed with.
What does NOT close this
- The measurement above. It settles which path to use and none of the work that follows.
- A member reaching a master after failover on the current rendering. That is Path A working,
which was never in doubt. - Deleting the lease-reading code without deciding the first bullet. The member is still handed
ak8s://address until that rendering changes, so removing the branch that reads it would break
the path that ships.
Where it came from
specs/2026-09-06-kv-cache-backend-high-availability.md - F4 ("The member's master address becomes
an HA entry") and T5. F4 assumed the scheme prefix was the mechanism; it never asked whether the
plain Service address already suffices. It does.
/area kv-cache
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with specs/2026-09-06-kv-cache-backend-high-availability.md, especially F4 and T5, then locate where the member address is rendered and where the existing non-HA Service address is assembled. Review the lease-reading path and decide whether it should be removed or made selectable, as the issue leaves that design choice open. Done means the member receives the leader Service address and the chosen lease-path behavior is documented and covered by the relevant checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100