HarperFast / HarperFast/harper-pro
`cluster_status.blobReplicationFailures` always reports 0 for copy-mode blob failures (observability gap)
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
During a full-copy resync with source-unavailable blobs, `recordBlobReplicationFailure()` fires (observed 14× in the repro) and the receiver logs `advancing the resume cursor past it` (7×), but `cluster_status` on **both** nodes reports `blobReplicationFailures: 0`. The log message itself cites `cluster_status.blobReplicationFailures` as the authoritative cumulative total — so an operator monitoring that field would never see blob-replication failures that ARE happening.
## Likely cause
The `auditStore` lookup in `clusterStatus.ts:47-52` finds nothing at query time (or reads a freshly-zeroed shared-buffer slot), so the metric-augmentation block is skipped. (harper-pro 282a0bcf)
## Severity
Low-medium — operator-facing observability gap, not data integrity. (The underlying advance-past behavior is separately tracked; this issue is specifically that the metric stays 0.)
## Repro
`integrationTests/cluster/qa-scratch/qa339-blob-resync-wedge.test.mjs` (from QA-339).
— from Harper exploratory QA (KrAIs)
Contributor guide
Research direction
Start with integrationTests/cluster/qa-scratch/qa339-blob-resync-wedge.test.mjs and inspect the auditStore lookup in clusterStatus.ts:47-52. Trace how recordBlobReplicationFailure() relates to the cluster_status metric during the full-copy resync. Done means cluster_status.blobReplicationFailures reflects the observed copy-mode blob failures on both nodes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs, typescript
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100