electric-sql / electric-sql/electric

Flaky timing-sensitive sync-service tests: ShapeCacheTest await/startup timeouts under CI load

Open
#4,712 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

## Summary

A small family of timing-sensitive tests fails intermittently in the `sync-service` CI workflow — hopping between Postgres matrix versions, passing on rerun with no code change, and burying real signal for PR authors. Two recent PRs (#4711 and its review) each spent effort separating these from genuine failures.

## Evidence

**`Electric.ShapeCacheTest` — "await_snapshot_start/4 should wait for consumer to come up"**
- Failed on pg15: https://github.com/electric-sql/electric/actions/runs/29346393580/job/87131112285 — `Task.await(..., 5300)` timeout
- Passed on rerun of the same commit (all matrices green)
- Failed on pg17 two commits later: https://github.com/electric-sql/electric/actions/runs/29348549229/job/87138530389 — same test, same timeout
- The failing branch's changes don't execute in this test (verified: the changed code is HTTP-serve-path only, and the branch's heavy `async: false` integration tests run *after* the async phase this test runs in — no load coupling)

**`Electric.ShapeCacheTest` — "get_or_create_shape_handle/2 against real db crashes when initial snapshot query fails to return data quickly enough"**
- Failed **on main**: https://github.com/electric-sql/electric/actions/runs/28917768297

**`Electric.Plug.TraceContextPlugTest` — "electric member is found among other tracestate members"**
- Failed **on main**: https://github.com/electric-sql/electric/actions/runs/28605359873

**`ShapeLogCollectorTest` — FlushTracker timing out in `Repatch.cleanup`**
- Reported during review of #4711 (first CI run of that branch)

## Common signature

- ~5s `Task.await` timeouts on tests that race consumer/snapshot startup
- Postgres connection churn in the surrounding logs (`tcp connect timeout`, `econnrefused`, `tcp recv (idle): closed`) suggesting the shared PG instance is briefly saturated by concurrent async tests
- Not reproducible locally on fast machines; version-hopping across the PG matrix

## Suggested directions

- Widen or parametrize the tight awaits in the affected tests (they're asserting *eventual* startup, not 5-second startup)
- Or address the underlying PG saturation (connection pool sizing / readiness in the test env)
- At minimum, tag the known-flaky set so authors and reviewers can distinguish them from real regressions without re-deriving this analysis

---
Filed while triaging CI on #4711, where these flakes twice muddied the signal.

Contributor guide

Open the contributing guide

Research direction

Start with the affected Electric.ShapeCacheTest, Electric.Plug.TraceContextPlugTest, and ShapeLogCollectorTest cases, focusing on their 5-second awaits and cleanup timing. Run the affected sync-service tests under the Postgres matrix and inspect the surrounding connection errors. Done means the tests remain meaningful, no longer flake under CI load, or are clearly tagged if the underlying saturation is not addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, github-actions, postgresql
Domain
ci-cd, databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.