hiero-ledger / hiero-ledger/hiero-consensus-node
Improve startup observability for key store loading and add key/cert correspondence check
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
## Goal
Improve the startup-time observability of `EnhancedKeyStoreLoader` so operators can confirm key store state from production logs and detect signing key / roster certificate mismatches before they manifest as gossip failures.
## Motivation
This work addresses follow-ups #8 and #9 from the [mainnet-9-nodes-down-after-upgrade #679 COE](https://www.notion.so/30c7c9ab2591808594a2e096cf372cef):
> 8. Platform: Cleanup/clarify/something to the key loading code. It's too confusing to understand under pressure. Maybe improve logging here as well (all logs are at the DEBUG level despite only being printed once at startup)
> 9. Platform: Add explicit checks on startup that compare the keys on disk to the gossip cert in the roster
During the incident, 9 mainnet nodes failed to establish gossip connections after an upgrade because their on-disk signing private keys did not pair with the gossip CA certificate the network had loaded from state. The startup logs gave no useful signal — nodes appeared healthy but stuck in `CHECKING`, with the failure surfacing only as `SSLHandshakeException: Received fatal alert: certificate_unknown` during the TLS handshake. Diagnosis required the DevOps team to manually run `openssl x509 -fingerprint -sha384` against every node's certificate to identify which keys were mismatched.
Today, key and certificate loading at startup logs at TRACE/DEBUG level — invisible in production. There is no startup-time check for whether the on-disk private key actually pairs with the cert in the roster.
## Proposed changes
1. Promote key and certificate loading logs from TRACE/DEBUG to INFO so they are visible in production.
2. Include the directory being scanned in the scan-start log, and a count summary (`sigKeysLoaded`, `sigCertsLoaded`) in the scan-completion log for operator confirmation.
3. Emit a WARN immediately when a node's roster cert is missing during scan, instead of silently deferring the signal to `verify()`.
4. Add a startup-time correspondence check in `verify()`: confirm that the on-disk signing private key pairs with the gossip CA certificate's public key in the roster. A mismatch produces a WARN that includes the cert SHA-384 fingerprint (matching `openssl x509 -fingerprint -sha384` output) so operators can cross-reference against the on-disk certificate.
5. The correspondence check is diagnostic, not fatal — the node continues to start so an existing-but-misconfigured node is not made worse off by the new behavior.
## Acceptance criteria
- INFO logs visible in production for key and cert loading operations
- WARN at scan time when a roster cert is missing for a local node
- WARN at verify time when the signing key on disk does not pair with the cert in the roster, including the cert SHA-384 fingerprint
- Node startup behavior unchanged: a detected mismatch produces a WARN but does not abort startup
Contributor guide
Assessment
This issue has not been assessed yet.