HarperFast / HarperFast/harper

Flaky integration test: blob.test.mjs 'Per-device-type LMDB database sharding' — schema missing from describe_all (shard 3/6)

Open
#1,557 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Symptom

`integrationTests/apiTests/blob.test.mjs` → **"Per-device-type LMDB database sharding"** fails intermittently in CI on **Integration Tests shard 3/6**, and passes on re-run with no code change. Two occurrences on the same PR (#1106) on 2026-07-01/02 alone — a PR whose diff is a pure formatter + test files that never touch blob/sharding code:

- [Run 28542021803, Node v22](https://github.com/HarperFast/harper/actions/runs/28542021803/job/84618295705): `AssertionError: doorlock schema missing` (in `all three device-type schemas are visible in describe_all`), then cascading `SQL queries target each device database independently: expected 200 "OK", got 404 "Not Found"`, with `HdbError: Table 'doorlock.DoorlockBlob' does not exist` in the Harper log. Passed on re-run.
- [Run 28558632042, Node v22 (re-run)](https://github.com/HarperFast/harper/actions/runs/28558632042/job/84675776989): same shape, different fixture — `AssertionError: sensor schema missing`, then the same 200→404 cascade. Passed on re-run.

Both times the same shard passed simultaneously on Bun, Node v24, Node v26, and Windows — the failure is timing-dependent, not version-deterministic (observed twice on v22, but the mechanism looks like scheduling luck rather than a v22 behavior).

## Shape of the race

The failing assertion is schema *visibility*: a device-type schema (`doorlock`, `sensor` — varies per occurrence) created during setup is missing from a subsequent `describe_all`, and SQL against that device database then 404s. The suite itself times out around ~30s (`Per-device-type LMDB database sharding (29992ms / 35174ms)`). This smells like create-schema/table completing on one worker thread before the metadata is visible to the thread serving the next request — same eventual-consistency family as #1497 (async ITC not awaited) and adjacent to #687 (create_table returns success before table is available). Possibly a condition-wait issue per the AGENTS.md guidance (prefer `waitFor(condition)` over fixed sleeps).

## Context

- The tests were added by #1195 ("Expand blob tests: multi-path blobPaths striping, per-device-type DB sharding", closed).
- Filed after the flake burned two re-run cycles on PR #1106 in a single day; it's currently the most frequent CI red we're hitting on unrelated PRs.

## Acceptance

- [ ] Root-cause the visibility race (or confirm it's a fixture-setup race) and fix, OR make the test's setup wait on actual schema visibility (`waitFor`-style condition on `describe_all` contents) rather than assuming synchronous visibility.
- [ ] No recurrence across ~20 consecutive shard-3 CI runs.

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with integrationTests/apiTests/blob.test.mjs and the “Per-device-type LMDB database sharding” test, then review the AGENTS.md guidance on condition waits. Reproduce the failure on Integration Tests shard 3/6 and inspect setup around describe_all visibility. Done means the race is fixed or setup waits for the expected schemas, with no recurrence across about 20 consecutive shard-3 CI runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.