hashgraph / hashgraph/solo-weaver
bug(bn/reset): storage clearing targets runner-host paths, not kind-node hostPath PVC locations — block files survive reset
- Dominant language
- Go
- Stars
- 3
- Forks
- 0
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 47
Description
## Summary
`solo-provisioner block node reset` reports clearing storage directories (e.g. `/tmp/bn/archive`, `/tmp/bn/live`) but in a **kind** cluster the actual block data is not removed. Block files survive the reset, so `firstAvailableBlock`/`lastAvailableBlock` remain non-zero after the pod restarts.
## Observed behaviour
From a GitHub Actions CI run against a kind cluster:
```
Clearing Block Node storage
Clearing storage directory path=/tmp/bn/archive
Clearing storage directory path=/tmp/bn/live
Clearing storage directory path=/tmp/bn/logs
Clearing storage directory path=/tmp/bn/plugins
Clearing storage directory path=/tmp/bn/application-state
All storage directories cleared successfully
```
After the BN pod restarts:
```
Started BlockNode Server : State=RUNNING HistoricBlockRange=0->4
```
`serverStatus` still reports `firstAvailableBlock: 0, lastAvailableBlock: 4`.
## Root cause
In kind, cluster "nodes" are Docker containers. hostPath PV paths (e.g. `/tmp/bn/archive`) resolve to paths **inside the Docker container acting as the kind node**, not on the runner host. solo-weaver's storage clearing runs on the machine executing the CLI (the runner), where those paths either do not exist or are empty. The `rm -rf` is a no-op and the PVC data is never touched.
`HistoricalBlockFacilityImpl` in the block node reports `firstAvailableBlock`/`lastAvailableBlock` by scanning actual block files on disk (not from a cached JSON file), so the surviving files are correctly reported as available.
The only reliable empty-state signal after reset is `nextExpectedBlock == uint64-max` (stream-publisher has no active session), because the publisher's in-memory state IS reset when the pod restarts.
## Expected behaviour
`block node reset` should clear block-storage PVC contents that are visible from inside the Kubernetes cluster, not from the runner host. This means the clearing must be performed **inside the cluster** — for example by:
- Running a temporary Kubernetes `Job` with the same PVCs mounted, executing `rm -rf /data/*`
- Or `kubectl exec`-ing into a short-lived pod with the PVCs
## Environment
- **solo-weaver version**: v0.26.0
- **Kubernetes distribution**: kind (Kubernetes IN Docker)
- **CI**: GitHub Actions (Ubuntu runner)
- **Block node version**: 0.39.x
## Workaround
Until this is fixed, callers can clear block storage from within the cluster before calling `reset`. For example (from a running BN pod before scale-down):
```bash
kubectl exec -n "${NAMESPACE}" "${POD}" -c block-node-server -- \
sh -c 'rm -rf /opt/hiero/block-node/data/live/* /opt/hiero/block-node/data/historic/*'
```
Note: clearing from a running pod while the BN is active risks in-flight IO corruption; it is safer to scale the pod down first, then clear via a temporary Job, then let solo-weaver scale back up.
Contributor guide
Assessment
This issue has not been assessed yet.