BrighterCommand / BrighterCommand/Brighter

gcp-ci: a persistent ~6-test timing tail, separate from #4321

Open
#4,324 0 comments 0 reactions 1 assignee Claimed by @iancooper View on GitHub
Dominant language
C#
Stars
2.5k
Forks
296
Avg merge
1d 11h
Merged PRs (30d)
21

Description

## Summary

`gcp-ci` shows a persistent tail of ~6 failing messaging-gateway tests that is **separate from** #4321. The tail is constant in size across runs but its membership shifts, and every failure has the same shape: `Expected: Not MT_NONE, Actual: MT_NONE` — a receive that waited and got nothing.

This was previously written off as a one-off flake. It is not: it has now reproduced on two consecutive runs.

## Evidence

| run | commit | total failures | FR-15 zero-delay (hard, #4321) | tail |
|---|---|---|---|---|
| [`34241244480`](https://github.com/BrighterCommand/Brighter/actions/runs/34241244480) (job `102147500855`) | `d6027e88e` | 10 | 4 | 6 |
| [`34330566094`](https://github.com/BrighterCommand/Brighter/actions/runs/34330566094) (job `102400301279`) | `f4b9a925b` | 10 | 4 | 6 |

**The 4 hard failures are #4321** — `When_requeuing_a_failed_message_with_zero_delay_should_redeliver_immediately` across `Pull` × `PullOrdering` × Reactor × Proactor. Those are reproducible and already tracked. **This issue is about the other 6.**

Tail membership on the two runs:

| test | run `34241244480` | run `34330566094` |
|---|---|---|
| `When_nacking_a_message_it_should_be_redelivered` | 2 | 2 |
| `When_rejecting_message_with_no_channels_configured_should_acknowledge_and_log` | 1 | 2 |
| `When_requeuing_a_failed_message_should_be_redelivered` | 2 | 1 |
| `When_sending_a_delayed_message_should_deliver_after_delay` | 1 | 1 |

Constant size (6), shifting membership, one failure shape.

## Reading

Consistent with **emulator/broker timing** rather than a per-behaviour conformance difference — which is why the right move is a separate flake issue rather than widening #4321 to name six more tests. Every test in the tail is one where a receive timed out.

Plausibly the same family as the DLQ/receive timing work tracked under #4240: where a test's outer wait loop only re-checks *between* provider calls, the effective bound is whatever the provider's internal retry budget is, not the bound the test documents. If that is the cause here, the fix is a timing fix, not a GCP behaviour fix.

## Suggested next steps

- [ ] Re-run `gcp-ci` a third time and confirm the tail is still ~6 with shifting membership
- [ ] Check whether the Pub/Sub emulator's ack-deadline / pull-window settings in `docker-compose-gcp.yaml` are tight relative to what the templates wait for
- [ ] Decide whether the canonical templates' hard-coded receive windows should be per-configuration (emulators and SQL-backed gateways have very different latency profiles)

## Not this

- Not #4321 — that is the 4 hard FR-15 failures, reproducible every run.
- Not a regression from the conformance-test branch: the tail appears identically on `d6027e88e` and `f4b9a925b`, and the intervening commits changed no generated file.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.