electric-sql / electric-sql/electric
Flaky timing-sensitive sync-service tests: ShapeCacheTest await/startup timeouts under CI load
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
## Summary
A small family of timing-sensitive tests fails intermittently in the `sync-service` CI workflow — hopping between Postgres matrix versions, passing on rerun with no code change, and burying real signal for PR authors. Two recent PRs (#4711 and its review) each spent effort separating these from genuine failures.
## Evidence
**`Electric.ShapeCacheTest` — "await_snapshot_start/4 should wait for consumer to come up"**
- Failed on pg15: https://github.com/electric-sql/electric/actions/runs/29346393580/job/87131112285 — `Task.await(..., 5300)` timeout
- Passed on rerun of the same commit (all matrices green)
- Failed on pg17 two commits later: https://github.com/electric-sql/electric/actions/runs/29348549229/job/87138530389 — same test, same timeout
- The failing branch's changes don't execute in this test (verified: the changed code is HTTP-serve-path only, and the branch's heavy `async: false` integration tests run *after* the async phase this test runs in — no load coupling)
**`Electric.ShapeCacheTest` — "get_or_create_shape_handle/2 against real db crashes when initial snapshot query fails to return data quickly enough"**
- Failed **on main**: https://github.com/electric-sql/electric/actions/runs/28917768297
**`Electric.Plug.TraceContextPlugTest` — "electric member is found among other tracestate members"**
- Failed **on main**: https://github.com/electric-sql/electric/actions/runs/28605359873
**`ShapeLogCollectorTest` — FlushTracker timing out in `Repatch.cleanup`**
- Reported during review of #4711 (first CI run of that branch)
## Common signature
- ~5s `Task.await` timeouts on tests that race consumer/snapshot startup
- Postgres connection churn in the surrounding logs (`tcp connect timeout`, `econnrefused`, `tcp recv (idle): closed`) suggesting the shared PG instance is briefly saturated by concurrent async tests
- Not reproducible locally on fast machines; version-hopping across the PG matrix
## Suggested directions
- Widen or parametrize the tight awaits in the affected tests (they're asserting *eventual* startup, not 5-second startup)
- Or address the underlying PG saturation (connection pool sizing / readiness in the test env)
- At minimum, tag the known-flaky set so authors and reviewers can distinguish them from real regressions without re-deriving this analysis
---
Filed while triaging CI on #4711, where these flakes twice muddied the signal.
Contributor guide
Research direction
Start with the affected Electric.ShapeCacheTest, Electric.Plug.TraceContextPlugTest, and ShapeLogCollectorTest cases, focusing on their 5-second awaits and cleanup timing. Run the affected sync-service tests under the Postgres matrix and inspect the surrounding connection errors. Done means the tests remain meaningful, no longer flake under CI load, or are clearly tagged if the underlying saturation is not addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir, github-actions, postgresql
- Domain
- ci-cd, databases, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100