hashgraph / hashgraph/solo-weaver

Add hardware benchmark subcommand for block node provisioning

Open
#591 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3
Forks
0
Avg merge
3d 2h
Merged PRs (30d)
46

Description

## Summary

`solo-provisioner` performs **static minimum-requirement checks** today (CPU cores, memory, storage capacity, SSD/HDD split, OS version, user/privilege setup) but does **not benchmark** the hardware it provisions.

The public [Hedera node requirements](https://docs.hedera.com/hedera/networks/mainnet/mainnet-nodes/node-requirements) already pin **concrete performance targets** (disk IOPS / throughput / latency, network bandwidth, CPU single-thread scores) — but today nothing in the provisioner measures actual hardware against them. To support **Block Node** hand-offs from Council Members, we want `solo-provisioner` to be the single tool a CM can run to **validate and benchmark** their hardware against a profile's performance envelope — not just its static spec.

This issue is a placeholder for requirements gathering. Once perf targets are firm, devs will run the benchmarks on real (or equivalent) hardware before designing the implementation.

> **Note:** the referenced public doc is the **Consensus Node** requirements page. If a separate Block Node requirements doc exists, that is the authoritative source for BN — see Open Questions.

## Current state

`pkg/hardware/` already gathers and validates:

- CPU core count
- Total + available memory (with a 512 MB system buffer)
- Storage capacity (SSD vs HDD split for `previewnet`)
- OS version (Ubuntu 18+, Debian 10+)
- User privileges / `weaver` user setup
- Disk **type** detection (SSD vs HDD categorization)

Surfaced today via:

```
solo-provisioner block node check --profile
solo-provisioner block node install --profile # check runs implicitly
```

> Closing additional **static-spec** gaps (e.g. NVMe vs generic SSD, ECC memory, RAID layout, supported OS/kernel list) is out of scope here and should be tracked as a sibling item that strengthens `check`. This issue is only about **performance benchmarking**.

## Gaps

Nothing in the provisioner exercises actual hardware **performance**. The published targets it could be measured against:

| Dimension | Published target (mainnet, from Hedera node-requirements doc) | Measured today? |
|---|---|---|
| **Disk I/O** — sequential | Seq Write 2–3 GB/s, Seq Read 3–6.2 GB/s | ❌ |
| **Disk I/O** — random | Rand Read 250k–1M IOPS (sync) / 500k–1M IOPS (AIO), Rand Write 100k–170k IOPS | ❌ |
| **Disk latency** | avg ≤200 µs | ❌ |
| **Network throughput** | ≥1 Gbps sustained per node; ≥200 Mbps per proxy connection | ❌ |
| **CPU single-thread** | Geekbench 6 single-core ≥1000 (rec ≥1500); Passmark single-thread ≥2300 mainnet (rec ≥2800) | ❌ |

## Proposed direction (for discussion)

Add a `benchmark` subcommand alongside `check` / `install`, profile-aware, that asserts measured performance against per-profile thresholds:

```
solo-provisioner block node benchmark --profile
```

Initial coverage:

- **Disk I/O** — IOPS / throughput / latency for sequential and random, read and write workloads. (`fio` is the industry reference.)
- **Network throughput** — sustained up/down. ([`librespeed-cli`](https://github.com/librespeed/speedtest-cli) is a strong candidate — open source and written in Go.)
- **CPU single-thread** — score comparable to the published Geekbench 6 / Passmark targets named in the Hedera docs.
- **System info capture** — hardware/kernel snapshot attached to the run artifact for traceability.

Per-profile thresholds would live in `pkg/hardware/` (e.g. extending `BaselineRequirements` with a `PerformanceTargets` section). Pass/fail aggregated through the same TUI output the existing `check` flow uses.

### Implementation preference: Go-native first, shell-out where unavoidable

Wherever a maintained Go library can produce results comparable to the de-facto industry tool, prefer it — keeps the binary self-contained, avoids runtime dependencies on the host, and makes results easier to test and parse. Realistically, parity is not always achievable on day one (e.g. `fio` is the reference for storage benchmarking and a pure-Go reimplementation would be a substantial undertaking), so an initial release may need to shell out to a small set of well-known tools and migrate to Go-native implementations incrementally. The network case looks favorable: `librespeed-cli` is itself a Go project and may be importable as a library rather than shelled out.

Decisions for each metric to make per implementation PR:

- Is there a Go library with comparable methodology and output to the reference tool?
- If shelling out, what's the install/version contract with the host, and how do we surface a clear error when the tool is missing?

## Open questions — input needed

### Functional

- [ ] **Block Node requirements source** — is there a BN-specific requirements doc, or are we extrapolating from the CN page? This determines the authoritative numbers for the table above.
- [ ] **Per-profile thresholds** — published numbers are mainnet-oriented; what are the targets for `local`, `perfnet`, `testnet`, `previewnet`?
- [ ] **Network latency** — is there a target RTT / jitter we want to assert (not specified in the public doc), and against which reference endpoint?
- [ ] **Memory bandwidth / latency** — is this something we want to benchmark at all? No published target exists today; including it would require defining one.
- [ ] **Pass/fail vs report-only** — should `benchmark` block `block node install` on failure, warn, or be opt-in?
- [ ] **Result destination** — file / JSON / TUI only, or also written to a known location for later collection?
- [ ] **Future scope** — should a corresponding `consensus node benchmark` follow the same pattern once BN lands?
- [ ] **Application-layer load testing** — is exercising the assembled node software (e.g. consensus-path load generators) ever in scope for a `benchmark` subcommand, or strictly a separate concern from the hardware micro-benchmarks proposed here?

### Non-functional

- [ ] **Runtime budget** — acceptable wall-clock for a CM-facing benchmark run (minutes vs hours)?
- [ ] **Privileges** — direct-I/O on raw devices vs file-backed; what we can safely do on a pre-handoff disk without destructive effects.
- [ ] **Idempotency / cleanup** — guarantees on temp files and any kernel/sysctl state left behind.
- [ ] **Reporting** — do we need machine-readable output (`--output json`) for downstream tooling?

## Out of scope (for this issue)

- Long-running (multi-hour) end-to-end load tests.
- Auto-remediation of failed benchmarks (e.g. tuning I/O scheduler, RAID reshape).
- Additional **static-spec** validation gaps (NVMe vs generic SSD, ECC, RAID, OS/kernel list). Tracked separately.

## References

- Hedera node minimum requirements (source of perf targets): https://docs.hedera.com/hedera/networks/mainnet/mainnet-nodes/node-requirements
- Existing static checks: `pkg/hardware/{spec.go,requirements.go,validation.go}`
- Existing command surface: `cmd/weaver/commands/block/` (check, install)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.