BrighterCommand / BrighterCommand/Brighter
gcp-ci: a persistent ~6-test timing tail, separate from #4321
- Dominant language
- C#
- Stars
- 2.5k
- Forks
- 296
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 21
Description
## Summary
`gcp-ci` shows a persistent tail of ~6 failing messaging-gateway tests that is **separate from** #4321. The tail is constant in size across runs but its membership shifts, and every failure has the same shape: `Expected: Not MT_NONE, Actual: MT_NONE` — a receive that waited and got nothing.
This was previously written off as a one-off flake. It is not: it has now reproduced on two consecutive runs.
## Evidence
| run | commit | total failures | FR-15 zero-delay (hard, #4321) | tail |
|---|---|---|---|---|
| [`34241244480`](https://github.com/BrighterCommand/Brighter/actions/runs/34241244480) (job `102147500855`) | `d6027e88e` | 10 | 4 | 6 |
| [`34330566094`](https://github.com/BrighterCommand/Brighter/actions/runs/34330566094) (job `102400301279`) | `f4b9a925b` | 10 | 4 | 6 |
**The 4 hard failures are #4321** — `When_requeuing_a_failed_message_with_zero_delay_should_redeliver_immediately` across `Pull` × `PullOrdering` × Reactor × Proactor. Those are reproducible and already tracked. **This issue is about the other 6.**
Tail membership on the two runs:
| test | run `34241244480` | run `34330566094` |
|---|---|---|
| `When_nacking_a_message_it_should_be_redelivered` | 2 | 2 |
| `When_rejecting_message_with_no_channels_configured_should_acknowledge_and_log` | 1 | 2 |
| `When_requeuing_a_failed_message_should_be_redelivered` | 2 | 1 |
| `When_sending_a_delayed_message_should_deliver_after_delay` | 1 | 1 |
Constant size (6), shifting membership, one failure shape.
## Reading
Consistent with **emulator/broker timing** rather than a per-behaviour conformance difference — which is why the right move is a separate flake issue rather than widening #4321 to name six more tests. Every test in the tail is one where a receive timed out.
Plausibly the same family as the DLQ/receive timing work tracked under #4240: where a test's outer wait loop only re-checks *between* provider calls, the effective bound is whatever the provider's internal retry budget is, not the bound the test documents. If that is the cause here, the fix is a timing fix, not a GCP behaviour fix.
## Suggested next steps
- [ ] Re-run `gcp-ci` a third time and confirm the tail is still ~6 with shifting membership
- [ ] Check whether the Pub/Sub emulator's ack-deadline / pull-window settings in `docker-compose-gcp.yaml` are tight relative to what the templates wait for
- [ ] Decide whether the canonical templates' hard-coded receive windows should be per-configuration (emulators and SQL-backed gateways have very different latency profiles)
## Not this
- Not #4321 — that is the 4 hard FR-15 failures, reproducible every run.
- Not a regression from the conformance-test branch: the tail appears identically on `d6027e88e` and `f4b9a925b`, and the intervening commits changed no generated file.
Contributor guide
Assessment
This issue has not been assessed yet.