hashgraph / hashgraph/solo-weaver
Failed block node install leaves an unrecoverable Helm release: reinstall and uninstall both error
- Dominant language
- Go
- Stars
- 3
- Forks
- 0
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 47
Description
## Problem
A failed block-node install leaves the deployment in a stuck state that the operator cannot recover from with solo-provisioner alone.
**Reproduction**
1. `sudo solo-provisioner block node install --plugins a,b` with non-existent plugin names `a`, `b`. This is a legitimate operator action — arbitrary plugins are intentionally supported. The install proceeds and the block-node pod crashloops because the chart's plugin resolver can't fetch `a.jar`/`b.jar`; the install ultimately fails. **(The crashloop itself is expected/desired — the operator chose plugins that don't exist. The problem is what happens next.)**
2. The failed attempt leaves a Helm release named `block-node` in a non-`deployed` status (`failed` / `pending-install`) that is never cleaned up.
3. Retrying `sudo solo-provisioner block node install` fails with:
```
helm.install_failed: failed to install chart, cause: cannot re-use a name that is still in use
```
4. `sudo solo-provisioner block node uninstall` refuses with:
```
block node is not installed; cannot uninstall
```
**Root cause**
solo-provisioner treats a block node as "installed" only when the Helm release status is `StatusDeployed`:
- `IsInstalled` (`pkg/helm/manager.go:508`) returns true only for `release.StatusDeployed`.
- The uninstall guard (`internal/bll/blocknode/uninstall_handler.go:40`) errors unless `currentState.BlockNodeState.ReleaseInfo.Status == release.StatusDeployed` (or `--force`).
A failed install leaves a release that *exists* but is not `Deployed`, so `install` (pure `helm install`) hits Helm's "name already in use" error, and `uninstall` refuses — even though the release is physically present and `helm uninstall block-node -n block-node` would clean it up.
> Note: `sudo solo-provisioner block node uninstall --force` already bypasses the guard and does clean up the failed release today. But it isn't discoverable from either error message, and plain `uninstall` shouldn't dead-end here.
## Proposed fix
1. **Enhance the install error.** When `block node install` fails with "cannot re-use a name that is still in use" / `helm.install_failed`, attach a resolution hint (per the repo's `StepFailureReport` resolution convention) telling the operator to run `sudo solo-provisioner block node uninstall` first to clean up the incomplete release before retrying. (Failure surfaces from `internal/workflows/steps/step_block_node.go` `installBlockNode`.)
2. **Make `uninstall` clean up a present-but-failed release.** In `UninstallHandler.BuildWorkflow` (`internal/bll/blocknode/uninstall_handler.go:40`), when the recorded status is not `StatusDeployed` but a Helm release actually exists (detect via the existing helm release/status lookup used by `IsInstalled` / `GetRelease`), proceed to run the `UninstallBlockNode` step (`helm uninstall block-node -n block-node`) instead of erroring. This lets plain `uninstall` recover the stuck state so the next install succeeds. Continue to error clearly only when no release exists at all.
## Acceptance
- [ ] Repro: `block node install --plugins a,b` (non-existent) → after the failed install a `block-node` Helm release exists in a non-`deployed` status.
- [ ] `block node uninstall` (no `--force`) detects the present-but-failed release and runs `helm uninstall`, leaving no `block-node` release.
- [ ] After that uninstall, `block node install` succeeds (no "name already in use").
- [ ] `block node install` failing on "name already in use" surfaces a resolution hint pointing to `block node uninstall`.
- [ ] `block node uninstall` when nothing is installed at all (no release present) still errors clearly — no regression.
## Context
Surfaced while demonstrating the intentional arbitrary-`--plugins` flexibility (block-node plugin work, #872). The failure mode is generic to any failed install (bad plugins are just the easiest trigger).
Contributor guide
Research direction
Read pkg/helm/manager.go around IsInstalled and GetRelease, then inspect internal/bll/blocknode/uninstall_handler.go and internal/workflows/steps/step_block_node.go around the install and uninstall entry points. Reproduce the failed plugin install and verify that plain uninstall removes an existing failed release, no-release uninstall still errors clearly, and the name-conflict failure includes the uninstall resolution hint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, helm, kubernetes
- Domain
- cli, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100