HarperFast / HarperFast/harper

Rows with undecodable keys silently end projected iterators and are unaddressable from the application layer

Open
#2,322 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## What happened

A production table (string primary key; URLs written by traffic-driven discovery) holds rows whose key attribute fails to decode. Three behaviors observed on 5.2.4, each independently problematic:

1. **A projected (`select`) resource-layer `search` silently ENDS its stream at such a row** — no error, no partial row. A cursor-chunked walk (`limit` N, resume from last key) reads the short chunk as end-of-range: measured, a maintenance walk reported **"finished" after 13,529 of 1,414,979 rows** on two nodes, and 6,704 on a third — the operator is told the range was fully covered.
2. **The ops-API search over the same range tolerates the row**: it is yielded mid-stream with the key attribute `null` (`get_attributes` projection and full-record reads both), and iteration continues. The two read paths disagree about the same data.
3. **The rows are unaddressable**: with no readable key there is no way to point-read, repair, or delete them through the API. They are permanent, and they differ per replica — one node's replica has 269 such rows in a 1.4M-row PK range while another node's replica of the same table has 0, so they appear to be born (or corrupted) per-node rather than replicated.

Possibly related: `[main/0] [warn]: Error serializing in stream Error: Next failed: Iterator not initialized` (extended-iterable `continueIteration`) observed on an affected node.

## Suspected origin

Discovery mints string PKs directly from crawler-supplied URLs; a URL percent-decoded into a JS string with lone surrogates / invalid UTF-8 would write a key that later fails decode. Not confirmed.

## Asks

- A projected iterator should not silently end at an undecodable row — yield it degraded (key null, like the ops path), or error, but never report a clean end mid-range.
- The two read paths should agree.
- Some supported way to enumerate and delete such rows (even just a scrub utility) — today they can only be counted by walking with the tolerant path and diffing.

## Notes

The application layer now works around this (skip-and-count + short-chunk verification before trusting end-of-range), so this is not blocking — but every naive chunked walk in any component hits behavior (1) and lies to its operator. Row positions, node names, and both read paths' outputs are available on request.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the projected resource-layer search with the ops-API search, including get_attributes and full-record reads, on a range containing an undecodable key. Inspect extended-iterable continueIteration and the stream serialization warning about “Iterator not initialized.” Done means projected iteration no longer reports a clean end mid-range, both read paths agree, and a supported way exists to enumerate and remove affected rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
api, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.