HarperFast / HarperFast/harper

Integration Tests on main failing ~47% of runs (2026-08-27/28): per-shard attribution, fixes landed, residual causes

Open
#2,379 0 comments 0 reactions 0 assignees View on GitHub
area:ci flaky-test
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Measurement

Integration Tests on `main`, 2026-08-27 00:11Z → 2026-08-28 13:25Z: **7 failures / 15 runs (47%)**. Widening to the last 30 runs (from 2026-08-25 22:33Z): 14 failures / 30.

Same-sha pass/fail pairs prove flake rather than broken commits: `9a24b055` failed on push ([33139925305](https://github.com/HarperFast/harper/actions/runs/33139925305)) and passed on the schedule ([33163451853](https://github.com/HarperFast/harper/actions/runs/33163451853)); `30db07ac` failed on push ([33026014429](https://github.com/HarperFast/harper/actions/runs/33026014429)) and passed on the schedule ([33062016523](https://github.com/HarperFast/harper/actions/runs/33062016523)).

## Every failing shard, attributed

Read from the job logs plus the `harper-server-logs-*` artifacts (`hdb.log` / `stderr.log`) of all 14 failing runs. Ten distinct failing test files collapse into six causes:

| cause | runs (shard) | signature | status |
|---|---|---|---|
| **A. Worker exits with code 0 before ready** → `Harper produced no startup output for 150000ms` (or SIGABRT on Node 22) | 6: [33090550550](https://github.com/HarperFast/harper/actions/runs/33090550550) (N26 2/6 `record-caching`), [33086796915](https://github.com/HarperFast/harper/actions/runs/33086796915) (N22 1/6 `eviction-index-orphan-removal-paths`, N22 6/6 `ops-port-isolation`), [33077733441](https://github.com/HarperFast/harper/actions/runs/33077733441) (N24 5/6 `record-caching-invalidate`), [33021521454](https://github.com/HarperFast/harper/actions/runs/33021521454) (N24 4/6 `record-caching-cross-worker`), [32980954252](https://github.com/HarperFast/harper/actions/runs/32980954252) (N24 3/6 `deleteUpdateRace`), [32980065492](https://github.com/HarperFast/harper/actions/runs/32980065492) (N22 6/6 `eviction-index-orphan-removal-paths`) | `Error: Worker (index N) exited with code 0 before reporting ready` at `socketRouter.ts:100` on a fresh-install multi-worker (`threads.count: 4`) start, then `Database not open` cascade in sibling workers | **fixed** by #2314 (2026-08-28 12:00Z), tracked in #2312 |
| **B. Acknowledged `addTo` increments lost** (`QA-431(5): 1 window(s) with lost counts`) | 4: [33139925305](https://github.com/HarperFast/harper/actions/runs/33139925305) (N26 1/6), [33026014429](https://github.com/HarperFast/harper/actions/runs/33026014429) (N24 5/6), [32980954252](https://github.com/HarperFast/harper/actions/runs/32980954252) (N22 4/6 + N24 4/6), [32980065492](https://github.com/HarperFast/harper/actions/runs/32980065492) (uWS 4/6) | `stored=49, thread 0` + `Deferring because its interrupted component deployment is still being prepared` as the most promising lead. It is **downstream, not a cause**: it appears in exactly one of the 14 failing runs (33139925305, Windows 6/6, two occurrences), and there it is `recoverInterruptedComponentExtractions` on worker restart tripping over the lock that cause C's wedged deploy is still holding (250 ms try-timeout, by design). On Linux it never appears. The `EPERM: operation not permitted, watch` uncaught exceptions are Windows-only noise present in passing runs too (12–16 per Windows job).

## Post-fix measurement

Every `main` run since #2314 merged: [33169340100](https://github.com/HarperFast/harper/actions/runs/33169340100), [33175354846](https://github.com/HarperFast/harper/actions/runs/33175354846), and three `workflow_dispatch` runs triggered for this measurement at 16:28Z — [33190074619](https://github.com/HarperFast/harper/actions/runs/33190074619), [33190079114](https://github.com/HarperFast/harper/actions/runs/33190079114), [33190089908](https://github.com/HarperFast/harper/actions/runs/33190089908) — **5/5 passing** (vs 8 of 15 passing in the window above). Small sample; with A and D gone the expected residual is cause B (~13% of runs on its own, until #2127 is fixed) plus C/E at a few percent each.

## Remaining work tracked elsewhere

- #2127 — lost `addTo` increments (P1; the one cause that is a product defect)
- #2273 — Windows deploy wedge (stage markers landed; awaiting recurrence)
- #2312 — worker pre-ready exit (fix landed; close on a clean week)
- `read_audit_log` `search_type: 'hash_value'` threw `TypeError: Cannot read properties of undefined (reading 'value')` for one key during the #2127 reproduction — untracked, small

Refs #1655.

— Claude Fable 5

Contributor guide

Open the contributing guide

Research direction

Start with the integration-test workflow and the named failure entry point at socketRouter.ts:100, then inspect the listed failing tests such as record-caching, redeploy-runtime-equivalence, and blob.test.mjs. The fixes and residual investigations are already tracked in #2127, #2273, and #2312; done means confirming the remaining causes are resolved through clean main runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, javascript, nodejs
Domain
backend, ci-cd, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.