HarperFast / HarperFast/harper
Flaky integration test: blob.test.mjs 'Per-device-type LMDB database sharding' — schema missing from describe_all (shard 3/6)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Symptom
`integrationTests/apiTests/blob.test.mjs` → **"Per-device-type LMDB database sharding"** fails intermittently in CI on **Integration Tests shard 3/6**, and passes on re-run with no code change. Two occurrences on the same PR (#1106) on 2026-07-01/02 alone — a PR whose diff is a pure formatter + test files that never touch blob/sharding code:
- [Run 28542021803, Node v22](https://github.com/HarperFast/harper/actions/runs/28542021803/job/84618295705): `AssertionError: doorlock schema missing` (in `all three device-type schemas are visible in describe_all`), then cascading `SQL queries target each device database independently: expected 200 "OK", got 404 "Not Found"`, with `HdbError: Table 'doorlock.DoorlockBlob' does not exist` in the Harper log. Passed on re-run.
- [Run 28558632042, Node v22 (re-run)](https://github.com/HarperFast/harper/actions/runs/28558632042/job/84675776989): same shape, different fixture — `AssertionError: sensor schema missing`, then the same 200→404 cascade. Passed on re-run.
Both times the same shard passed simultaneously on Bun, Node v24, Node v26, and Windows — the failure is timing-dependent, not version-deterministic (observed twice on v22, but the mechanism looks like scheduling luck rather than a v22 behavior).
## Shape of the race
The failing assertion is schema *visibility*: a device-type schema (`doorlock`, `sensor` — varies per occurrence) created during setup is missing from a subsequent `describe_all`, and SQL against that device database then 404s. The suite itself times out around ~30s (`Per-device-type LMDB database sharding (29992ms / 35174ms)`). This smells like create-schema/table completing on one worker thread before the metadata is visible to the thread serving the next request — same eventual-consistency family as #1497 (async ITC not awaited) and adjacent to #687 (create_table returns success before table is available). Possibly a condition-wait issue per the AGENTS.md guidance (prefer `waitFor(condition)` over fixed sleeps).
## Context
- The tests were added by #1195 ("Expand blob tests: multi-path blobPaths striping, per-device-type DB sharding", closed).
- Filed after the flake burned two re-run cycles on PR #1106 in a single day; it's currently the most frequent CI red we're hitting on unrelated PRs.
## Acceptance
- [ ] Root-cause the visibility race (or confirm it's a fixture-setup race) and fix, OR make the test's setup wait on actual schema visibility (`waitFor`-style condition on `describe_all` contents) rather than assuming synchronous visibility.
- [ ] No recurrence across ~20 consecutive shard-3 CI runs.
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start with integrationTests/apiTests/blob.test.mjs and the “Per-device-type LMDB database sharding” test, then review the AGENTS.md guidance on condition waits. Reproduce the failure on Integration Tests shard 3/6 and inspect setup around describe_all visibility. Done means the race is fixed or setup waits for the expected schemas, with no recurrence across about 20 consecutive shard-3 CI runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- databases, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100