OpenBao post-install init job fails on clean install and cannot recover on retry
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Problem
A clean helm install of the self-managed OpenBao chart can leave the release
in a failed state. The post-install openbao-server-initialize-cluster Job ends
in BackoffLimitExceeded and the raft cluster is never fully bootstrapped.
Two independent defects in deploy/helm/openbao/**/deploy.sh combine to make
this unrecoverable:
-
Raft join races leader election. After unsealing
openbao-server-0,
unseal_clusterwaits a fixedsleep 5and then immediately runs
bao operator raft joinfrom the peers. If the primary has not yet won
leader election it cannot serve raft challenges, and the join fails with
HTTP 500 ... failed to join raft cluster: failed to get raft challenge.
There is no retry, so a single unlucky attempt fails the whole step. -
Initialization is not idempotent, so the Job's own retries cannot recover.
The Job has nobackoffLimit, so it takes the Kubernetes default of 6 and
creates up to 7 pods per install. The "already initialized" precheck only
runs when the script is invoked withinstall_method=script, but the Job
invokes it withhelm, soinitialize_clusterrunsbao operator init
unconditionally. Once one attempt has initialized the cluster, every
subsequent attempt fails withHTTP 400 Vault is already initializedand
exits before reaching the raft bootstrap that the earlier attempt left
unfinished.
The result is that the retry budget is spent without progress, and the install
has to be cleaned up and started over by hand.
Expected behavior
A single helm install on a clean cluster should bootstrap the OpenBao raft
cluster without manual intervention, and the Job's automatic retries should be
able to finish work a previous attempt started rather than failing on state
that attempt created.
Notes
Both deploy/helm/openbao/deploy.sh and
deploy/helm/openbao/helm/scripts/deploy.sh carry this logic and need to stay
in sync.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing deploy/helm/openbao/deploy.sh with deploy/helm/openbao/helm/scripts/deploy.sh, focusing on unseal_cluster, initialize_cluster, and the post-install Job behavior. Reproduce or inspect a clean helm install and its retry sequence. Done means a clean install bootstraps the raft cluster and retries can continue after partial initialization without manual cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, shell
- Domain
- devops, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100