BrighterCommand / BrighterCommand/Brighter
dynamo-ci: CausationTrackingOutboxTests reads an eventually-consistent GSI and asserts it is up to date
- Dominant language
- C#
- Stars
- 2.5k
- Forks
- 296
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 21
Description
`dynamo-ci` is the least reliable job on `master`, and every failure in the recent window is the same test class. This is a **test** defect rather than a product one: the assertions require an immediacy that a DynamoDB Global Secondary Index cannot promise.
## The census — 5 failures in 13 `master` runs, all the same class
```text
2026-09-12T14:14:54Z run 34698723388 failure CausationTrackingOutboxTests
2026-09-12T09:14:33Z run 34685324878 success
2026-09-11T15:52:07Z run 34618666177 success
2026-09-11T14:14:18Z run 34608934318 failure CausationTrackingOutboxTests
2026-09-11T09:13:05Z run 34583067399 success
2026-09-10T14:27:57Z run 34489230421 success
2026-09-10T08:00:04Z run 34452809953 success
2026-09-10T07:37:47Z run 34450911909 failure CausationTrackingOutboxTests
2026-09-09T20:12:04Z run 34399594080 success
2026-09-08T10:12:10Z run 34214176968 success
2026-09-07T20:32:17Z run 34159782273 success
2026-09-07T20:30:03Z run 34159638104 failure CausationTrackingOutboxTests
2026-09-07T16:50:22Z run 34145065842 failure CausationTrackingOutboxTests
```
**5 of 5 `dynamo-ci` failures are `Outbox.Causation.CausationTrackingOutboxTests`**, across five days, in both the `Paramore.Brighter.DynamoDB.Tests` and `Paramore.Brighter.DynamoDB.V4.Tests` assemblies. The named tests are `When_replaying_causation_on_outbox_should_clear_dispatch_state`, its `_async` twin, and `When_replaying_causation_for_messages_deposited_in_bulk_should_clear_dispatch_state`.
The failure is always the same shape:
```text
Assert.DoesNotContain() Failure: Item found in collection
```
## Why it cannot be made reliable as written
`tests/Paramore.Brighter.Base.Test/Outbox/CausationTrackingOutboxBaseTests.cs:85-93` writes and then immediately reads back, asserting the write is already visible:
```csharp
Outbox.MarkDispatched(firstWithA.Id, contextA, dispatchedAt);
Outbox.MarkDispatched(secondWithA.Id, contextA, dispatchedAt);
Outbox.MarkDispatched(messageWithB.Id, contextB, dispatchedAt);
// all three start dispatched, so none are outstanding
var outstandingBefore = Outbox.OutstandingMessages(TimeSpan.Zero, contextA).Select(m => m.Id).ToArray();
Assert.DoesNotContain(firstWithA.Id, outstandingBefore);
Assert.DoesNotContain(secondWithA.Id, outstandingBefore);
```
On DynamoDB, `OutstandingMessages` is a **GSI query** — `src/Paramore.Brighter.Outbox.DynamoDB.V4/DynamoDbOutbox.cs:835-836` and `:937-938`:
```csharp
IndexName = _configuration.OutstandingAllTopicsIndexName,
ConsistentRead = false,
```
**`ConsistentRead = false` is not a choice that can be reversed here.** DynamoDB does not support strongly consistent reads on a Global Secondary Index at all — requesting one is an error. GSI propagation after a write is asynchronous and, while usually within milliseconds, is not bounded. So a message marked dispatched can still be returned by the outstanding index for a short window, and `Assert.DoesNotContain` finds it.
The same class reads consistently where it *can* — `:99` uses `ConsistentRead = true` for a primary-key get — which is the contrast that makes the diagnosis concrete rather than speculative.
## This is not the product misbehaving
Eventual consistency on the outstanding index is inherent to the store, and the Outbox Sweeper tolerates it by design: a message that appears outstanding when it is not is re-swept, and the Inbox de-duplicates. The defect is that the test asserts an immediacy the store never offered.
## Suggested fix
Something that expresses "eventually", rather than a sleep:
- Poll `OutstandingMessages` until the expected state is reached or a generous timeout expires, and assert on the settled result. This keeps the test meaningful — it still fails if the state never becomes correct — without asserting a propagation deadline.
- Or override the pre-Act assertion for the DynamoDB implementations, since `outstandingBefore` at `:91-93` is arranging rather than asserting the behaviour under test. The assertions that matter are after `ReplayCausation`.
The bulk variant fails the same way and for the same reason.
## Scope
Found while working on #4343, which is unrelated — the failures are on `master` without it, which is how it was attributed rather than guessed at. Not filed as a flake to be re-run: the cause is identified and it will keep recurring at roughly the observed rate until the assertions change.
Contributor guide
Assessment
This issue has not been assessed yet.