HarperFast / HarperFast/harper-pro

`cluster_status.blobReplicationFailures` always reports 0 for copy-mode blob failures (observability gap)

Open
#502 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary
During a full-copy resync with source-unavailable blobs, `recordBlobReplicationFailure()` fires (observed 14× in the repro) and the receiver logs `advancing the resume cursor past it` (7×), but `cluster_status` on **both** nodes reports `blobReplicationFailures: 0`. The log message itself cites `cluster_status.blobReplicationFailures` as the authoritative cumulative total — so an operator monitoring that field would never see blob-replication failures that ARE happening.

## Likely cause
The `auditStore` lookup in `clusterStatus.ts:47-52` finds nothing at query time (or reads a freshly-zeroed shared-buffer slot), so the metric-augmentation block is skipped. (harper-pro 282a0bcf)

## Severity
Low-medium — operator-facing observability gap, not data integrity. (The underlying advance-past behavior is separately tracked; this issue is specifically that the metric stays 0.)

## Repro
`integrationTests/cluster/qa-scratch/qa339-blob-resync-wedge.test.mjs` (from QA-339).

— from Harper exploratory QA (KrAIs)

Contributor guide

Open the contributing guide

Research direction

Start with integrationTests/cluster/qa-scratch/qa339-blob-resync-wedge.test.mjs and inspect the auditStore lookup in clusterStatus.ts:47-52. Trace how recordBlobReplicationFailure() relates to the cluster_status metric during the full-copy resync. Done means cluster_status.blobReplicationFailures reflects the observed copy-mode blob failures on both nodes.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs, typescript
Domain
backend, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.