hashgraph / hashgraph/solo-weaver
feat(block-node): place operator-declared files onto the storage tree before the pod starts
- Dominant language
- Go
- Stars
- 3
- Forks
- 0
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 47
Description
## Problem
Council members resetting a mainnet block node need to stage a file — concretely
`rsa-bootstrap-roster.json` — into the application-state directory on the PV *before* the
block node starts. There is no way to do that as part of a weaver command today.
`block node reset` scales the pod down, wipes the data directories and brings it back up.
Anything copied after the command returns races the BN's startup, and because reset wipes on
every run, a file staged by hand once disappears at the next reset. The same holds on `install`
— `SetupBlockNode` creates the directories and installs the chart back to back — and on any
path that rebuilds the tree through `RecreateBlockNodeStorage`.
`--no-scale-up` (#1153, merged in #1156) makes a manual workaround possible: stop the reset
halfway, copy the file by hand, bring the node back up. That works, but it is manual,
unrecorded, and has to be repeated correctly on every reset by every council member on their own
host — exactly the class of step that goes wrong during a coordinated mainnet reset window.
## Proposed fix
Declare the files in the config file and have weaver (re-)place them every time the storage tree
is created or wiped.
```yaml
blockNode:
files:
- source: /opt/solo/staging/rsa-bootstrap-roster.json
destination: applicationState/rsa-bootstrap-roster.json
mode: "0640"
```
The list is a desired-state declaration, not a one-shot copy: after any weaver command that
touches the storage tree, the declared files are present with the declared ownership and mode.
### Placement points
`ResetBlockNode(deployedStorage, inputs)` (as of #1157) gives reset a single unambiguous slot —
after everything that can create or wipe the tree, before the pod can read it:
```
purgeBlockNodeStorageSteps(deployedIns) // clears at the DEPLOYED paths
RecreateBlockNodeStorage(inputs) // --purge-storage only; new paths
<-- place files here, against inputs.Storage
scaleUpBlockNode + waitForBlockNode // unless --no-scale-up
```
| Command | Insert after | Insert before |
|---|---|---|
| `install` | `setupBlockNodeStorage` | `installBlockNode` |
| `reset` | `RecreateBlockNodeStorage` when purging, else the purge steps | `scaleUpBlockNode` |
| `reconfigure` / `upgrade` with `--purge-storage` or `--with-reset` | `RecreateBlockNodeStorage` | `UpgradeBlockNode` |
The invariant is "re-assert after anything that creates or wipes the tree", never "copy once at
install". Placing before the pod starts is what makes the file visible to the BN's first read;
placing after every wipe is what stops reset from silently undoing it.
Destinations resolve against `inputs.Storage` — the paths the operator wants next — never
`deployedStorage`. Under `reset --purge-storage` and `reconfigure --purge-storage` those differ,
and the files must land in the freshly recreated tree.
### Destination addressing
Resolve `destination` through a storage-key prefix (`applicationState`, `live`, `archive`, `log`,
`plugins`, `verification`) against the effective storage paths, and validate the result with
`sanity.ValidatePathWithinBase`.
Raw absolute host paths are deliberately not the interface. They break under base-path mode, they
go stale when `--purge-storage` moves the tree, and a typo writes outside the managed directories
with no guard.
### Ownership, mode and write semantics
- Own the file `hedera:hedera` via `fsManager.WriteOwnerByName(config.HederaUserName(),
config.HederaGroupName(), …)` and set the mode with `WritePermissions`, matching what
`SetupStorage` already does for the directories. A root-owned file the BN cannot read is the
most likely failure mode here.
- Write atomically, with the temp file on the destination's own filesystem (the PV) — a temp in a
staging dir or `/tmp` fails the final rename with `EXDEV` across filesystems.
- Overwrite unconditionally. Reset wipes the tree anyway, and "declared state" is the model — a
skip-if-present rule would make the result depend on history.
### Scope
In scope: a local host path as `source`, for the block node only.
Out of scope: URL downloads, checksum verification and archive extraction — see the first open
question below.
## Open question 1 — reuse `external-files.yaml`, or build a second mechanism?
This overlaps with the consensus external-files work, and the overlap is already half-built.
`pkg/manifests/external_files.go` is merged and provides:
* `ExternalFile{URL, Algorithm, Checksum, ContentType, Destination, Optional, Phase}`
* `allowedDestinationPrefixes` — a **closed set** of marker prefixes (`HAPIAPP_DIR`,
`SOLO_PROVISIONER_DIR`), exposed via `AllowedDestinationPrefixes()`
* `validateDestinationPrefix` — rejects anything not starting with a recognised marker followed
by `/`
* schema versioning via `Header` / `MigrateToLatest` / `SupportedVersions`
What does **not** exist is the execution half — download, atomic write, ownership. #537 and #538
are both still open. So the schema and validation are done; the placement is not. The design
sketched above *is* #538's design, which means shipping it under `blockNode.files` writes the
placement engine twice and then maintains two destination vocabularies, two validators and two
schema-migration paths.
**(a) Extend the manifest.** Add block-node storage markers to the existing closed set, resolve
them from the effective storage paths, and let a block node consume an `external-files.yaml`.
Gets checksums, `optional:`, schema versioning and the phase model for free. Costs: `Phase` is
consensus-freeze-shaped (`phase.install` only permits `freeze`) and needs a block-node-meaningful
value or an explicit exemption; and the manifest is URL-first, so a local-path source needs
adding.
**(b) Keep `blockNode.files` separate.** Simpler to ship, no consensus coupling, no phase
semantics to bend. Costs: a second placement engine, and the near-certainty that checksums and
`optional:` get asked for later anyway.
Worth deciding before any code is written — the two diverge on day one.
## Open question 2 — the list has to survive a run with no `--config`
A list that lives only in `config.yaml` silently un-asserts itself on any run that does not pass
`-c`:
```
install -c prod.yaml # roster placed
reset # no -c → nothing declared → tree wiped, nothing placed
```
That is precisely the failure this issue exists to prevent, and it is the likely real-world
sequence: the council member running the reset is not necessarily the person who wrote the config
file. #1160 sharpens this rather than solving it — by establishing that a config file only counts
when explicitly passed (`config.File()`), it makes the empty case the default one.
The repo already solved this shape for the host firewall, and that precedent should be copied
rather than rediscovered:
* `patchMachineFirewallFromConfig` records the resolved `config.Get().Host` into
`MachineState.Firewall`, whose doc comment says it exists so reconfigure/upgrade can re-assert
"without the operator re-supplying `--mgmt-cidrs`"
* `SeedHostFirewallFromState` reads it back on `upgrade`, which exposes no firewall flags
* `applyPersistedFirewallContent` fills only the fields config left empty, so an explicit `-c`
still wins
So this needs three pieces: persist the resolved list into `BlockNodeState`, seed from it when no
file is supplied, and let an explicit `-c` override — which #1160 already grants for `reset`
(`promotesConfigFile` includes `ActionReset`; only `uninstall` is excluded).
**The sub-decision:** on a list, "not specified" and "specified as empty" are indistinguishable.
If a run supplies `-c` with no `blockNode.files` key, does that mean "place nothing, drop the
persisted list" or "I said nothing about files, keep what is on record"? Wrong in the first
direction and a routine reconfigure silently stops placing the roster; wrong in the second and an
operator can never remove a file from the list. The firewall code hit this and worked around it
with an explicit content probe, because `HostConfig.Disabled`'s zero value cannot distinguish
"enabled" from "never configured". This needs the same probe or an explicit `files: []` sentinel.
## Risks
* **`viper.UnmarshalExact`.** `config.Initialize` rejects unknown keys outright, so any config
carrying `blockNode.files` fails to parse on a weaver build that predates the schema. The schema
has to land before the key appears in any council member's config file.
## Acceptance
- [ ] `blockNode.files` is accepted in the config schema with `source`, `destination` and an
optional `mode`; an unknown or malformed entry fails validation with a clear message.
- [ ] `destination` resolves through a storage-key prefix against the effective storage paths and
is rejected when it escapes the resolved storage directory.
- [ ] A declared file is present, owned `hedera:hedera`, with the declared mode, after
`block node install`, before the chart is installed.
- [ ] A declared file is present before the pod is scaled back up by `block node reset`, and
survives repeated resets without operator action.
- [ ] `reset --purge-storage` and `reconfigure --purge-storage` place the files in the recreated
tree at the **new** paths, not the deployed ones.
- [ ] `reset --no-scale-up` leaves the declared files in place on a stopped node.
- [ ] **A declared file is re-placed on a run that supplies no `--config`**, from the persisted
record rather than from the config file.
- [ ] An explicit `--config` overrides the persisted list; the "omitted key" vs "empty list"
distinction behaves as decided in open question 2 and is pinned by a test.
- [ ] A missing `source` fails the step with the path in the error rather than silently skipping.
- [ ] The write is atomic and the temp file is created on the destination filesystem.
- [ ] Unit tests pin the step's position in the workflow for install, reset, `reset
--purge-storage`, and the two upgrade/reconfigure purge paths.
- [ ] `docs/reference/configuration.md` documents the section; `docs/commands/block-node.md` notes
that declared files are re-placed on every reset.
### Related Issues
* Relates to #1153 / #1156 — `--no-scale-up` is the manual workaround this replaces, and it is the
flag the "stopped node" acceptance criterion exercises
* Relates to #1154 / #1157 — `RecreateBlockNodeStorage` and the `deployedStorage` vs
`inputs.Storage` split define where the placement step goes
* Relates to #1155 / #1160 — `config.File()` is the "explicitly supplied config" gate this needs,
and `promotesConfigFile` already covers `reset`
* Prior art: #537, #538, #1084 — `external-files.yaml` download, atomic placement and archives
Contributor guide
Research direction
Start with pkg/manifests/external_files.go and the existing block-node entry points SetupBlockNode, ResetBlockNode, RecreateBlockNodeStorage, and config.File(). Resolve the open design choices before implementation, then trace install, reset, and purge workflows to identify their placement points. Done means the schema, persistence behavior, path and write validation, workflow tests, and the two named documentation updates satisfy the acceptance checklist.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- cli, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100