hashgraph / hashgraph/solo-weaver

bug(bn/reset): storage clearing targets runner-host paths, not kind-node hostPath PVC locations — block files survive reset

Open
#935 0 comments 0 reactions 1 assignee Claimed by @alex-au View on GitHub
Dominant language
Go
Stars
3
Forks
0
Avg merge
3d 5h
Merged PRs (30d)
47

Description

## Summary

`solo-provisioner block node reset` reports clearing storage directories (e.g. `/tmp/bn/archive`, `/tmp/bn/live`) but in a **kind** cluster the actual block data is not removed. Block files survive the reset, so `firstAvailableBlock`/`lastAvailableBlock` remain non-zero after the pod restarts.

## Observed behaviour

From a GitHub Actions CI run against a kind cluster:

```
Clearing Block Node storage
Clearing storage directory path=/tmp/bn/archive
Clearing storage directory path=/tmp/bn/live
Clearing storage directory path=/tmp/bn/logs
Clearing storage directory path=/tmp/bn/plugins
Clearing storage directory path=/tmp/bn/application-state
All storage directories cleared successfully
```

After the BN pod restarts:

```
Started BlockNode Server : State=RUNNING HistoricBlockRange=0->4
```

`serverStatus` still reports `firstAvailableBlock: 0, lastAvailableBlock: 4`.

## Root cause

In kind, cluster "nodes" are Docker containers. hostPath PV paths (e.g. `/tmp/bn/archive`) resolve to paths **inside the Docker container acting as the kind node**, not on the runner host. solo-weaver's storage clearing runs on the machine executing the CLI (the runner), where those paths either do not exist or are empty. The `rm -rf` is a no-op and the PVC data is never touched.

`HistoricalBlockFacilityImpl` in the block node reports `firstAvailableBlock`/`lastAvailableBlock` by scanning actual block files on disk (not from a cached JSON file), so the surviving files are correctly reported as available.

The only reliable empty-state signal after reset is `nextExpectedBlock == uint64-max` (stream-publisher has no active session), because the publisher's in-memory state IS reset when the pod restarts.

## Expected behaviour

`block node reset` should clear block-storage PVC contents that are visible from inside the Kubernetes cluster, not from the runner host. This means the clearing must be performed **inside the cluster** — for example by:

- Running a temporary Kubernetes `Job` with the same PVCs mounted, executing `rm -rf /data/*`
- Or `kubectl exec`-ing into a short-lived pod with the PVCs

## Environment

- **solo-weaver version**: v0.26.0
- **Kubernetes distribution**: kind (Kubernetes IN Docker)
- **CI**: GitHub Actions (Ubuntu runner)
- **Block node version**: 0.39.x

## Workaround

Until this is fixed, callers can clear block storage from within the cluster before calling `reset`. For example (from a running BN pod before scale-down):

```bash
kubectl exec -n "${NAMESPACE}" "${POD}" -c block-node-server -- \
sh -c 'rm -rf /opt/hiero/block-node/data/live/* /opt/hiero/block-node/data/historic/*'
```

Note: clearing from a running pod while the BN is active risks in-flight IO corruption; it is safer to scale the pod down first, then clear via a temporary Job, then let solo-weaver scale back up.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.