NVIDIA / NVIDIA/nvcf

OpenBao post-install init job fails on clean install and cannot recover on retry

Open
#820 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Problem

A clean helm install of the self-managed OpenBao chart can leave the release
in a failed state. The post-install openbao-server-initialize-cluster Job ends
in BackoffLimitExceeded and the raft cluster is never fully bootstrapped.

Two independent defects in deploy/helm/openbao/**/deploy.sh combine to make
this unrecoverable:

  1. Raft join races leader election. After unsealing openbao-server-0,
    unseal_cluster waits a fixed sleep 5 and then immediately runs
    bao operator raft join from the peers. If the primary has not yet won
    leader election it cannot serve raft challenges, and the join fails with
    HTTP 500 ... failed to join raft cluster: failed to get raft challenge.
    There is no retry, so a single unlucky attempt fails the whole step.

  2. Initialization is not idempotent, so the Job's own retries cannot recover.
    The Job has no backoffLimit, so it takes the Kubernetes default of 6 and
    creates up to 7 pods per install. The "already initialized" precheck only
    runs when the script is invoked with install_method=script, but the Job
    invokes it with helm, so initialize_cluster runs bao operator init
    unconditionally. Once one attempt has initialized the cluster, every
    subsequent attempt fails with HTTP 400 Vault is already initialized and
    exits before reaching the raft bootstrap that the earlier attempt left
    unfinished.

The result is that the retry budget is spent without progress, and the install
has to be cleaned up and started over by hand.

Expected behavior

A single helm install on a clean cluster should bootstrap the OpenBao raft
cluster without manual intervention, and the Job's automatic retries should be
able to finish work a previous attempt started rather than failing on state
that attempt created.

Notes

Both deploy/helm/openbao/deploy.sh and
deploy/helm/openbao/helm/scripts/deploy.sh carry this logic and need to stay
in sync.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing deploy/helm/openbao/deploy.sh with deploy/helm/openbao/helm/scripts/deploy.sh, focusing on unseal_cluster, initialize_cluster, and the post-install Job behavior. Reproduce or inspect a clean helm install and its retry sequence. Done means a clean install bootstraps the raft cluster and retries can continue after partial initialization without manual cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, shell
Domain
devops, distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.