BrighterCommand / BrighterCommand/Brighter

dynamo-ci: CausationTrackingOutboxTests reads an eventually-consistent GSI and asserts it is up to date

Open
#4,350 1 comment 0 reactions 1 assignee Claimed by @iancooper View on GitHub
.NET 0 - Backlog V10.X
Dominant language
C#
Stars
2.5k
Forks
296
Avg merge
1d 11h
Merged PRs (30d)
21

Description

`dynamo-ci` is the least reliable job on `master`, and every failure in the recent window is the same test class. This is a **test** defect rather than a product one: the assertions require an immediacy that a DynamoDB Global Secondary Index cannot promise.

## The census — 5 failures in 13 `master` runs, all the same class

```text
2026-09-12T14:14:54Z run 34698723388 failure CausationTrackingOutboxTests
2026-09-12T09:14:33Z run 34685324878 success
2026-09-11T15:52:07Z run 34618666177 success
2026-09-11T14:14:18Z run 34608934318 failure CausationTrackingOutboxTests
2026-09-11T09:13:05Z run 34583067399 success
2026-09-10T14:27:57Z run 34489230421 success
2026-09-10T08:00:04Z run 34452809953 success
2026-09-10T07:37:47Z run 34450911909 failure CausationTrackingOutboxTests
2026-09-09T20:12:04Z run 34399594080 success
2026-09-08T10:12:10Z run 34214176968 success
2026-09-07T20:32:17Z run 34159782273 success
2026-09-07T20:30:03Z run 34159638104 failure CausationTrackingOutboxTests
2026-09-07T16:50:22Z run 34145065842 failure CausationTrackingOutboxTests
```

**5 of 5 `dynamo-ci` failures are `Outbox.Causation.CausationTrackingOutboxTests`**, across five days, in both the `Paramore.Brighter.DynamoDB.Tests` and `Paramore.Brighter.DynamoDB.V4.Tests` assemblies. The named tests are `When_replaying_causation_on_outbox_should_clear_dispatch_state`, its `_async` twin, and `When_replaying_causation_for_messages_deposited_in_bulk_should_clear_dispatch_state`.

The failure is always the same shape:

```text
Assert.DoesNotContain() Failure: Item found in collection
```

## Why it cannot be made reliable as written

`tests/Paramore.Brighter.Base.Test/Outbox/CausationTrackingOutboxBaseTests.cs:85-93` writes and then immediately reads back, asserting the write is already visible:

```csharp
Outbox.MarkDispatched(firstWithA.Id, contextA, dispatchedAt);
Outbox.MarkDispatched(secondWithA.Id, contextA, dispatchedAt);
Outbox.MarkDispatched(messageWithB.Id, contextB, dispatchedAt);

// all three start dispatched, so none are outstanding
var outstandingBefore = Outbox.OutstandingMessages(TimeSpan.Zero, contextA).Select(m => m.Id).ToArray();
Assert.DoesNotContain(firstWithA.Id, outstandingBefore);
Assert.DoesNotContain(secondWithA.Id, outstandingBefore);
```

On DynamoDB, `OutstandingMessages` is a **GSI query** — `src/Paramore.Brighter.Outbox.DynamoDB.V4/DynamoDbOutbox.cs:835-836` and `:937-938`:

```csharp
IndexName = _configuration.OutstandingAllTopicsIndexName,
ConsistentRead = false,
```

**`ConsistentRead = false` is not a choice that can be reversed here.** DynamoDB does not support strongly consistent reads on a Global Secondary Index at all — requesting one is an error. GSI propagation after a write is asynchronous and, while usually within milliseconds, is not bounded. So a message marked dispatched can still be returned by the outstanding index for a short window, and `Assert.DoesNotContain` finds it.

The same class reads consistently where it *can* — `:99` uses `ConsistentRead = true` for a primary-key get — which is the contrast that makes the diagnosis concrete rather than speculative.

## This is not the product misbehaving

Eventual consistency on the outstanding index is inherent to the store, and the Outbox Sweeper tolerates it by design: a message that appears outstanding when it is not is re-swept, and the Inbox de-duplicates. The defect is that the test asserts an immediacy the store never offered.

## Suggested fix

Something that expresses "eventually", rather than a sleep:

- Poll `OutstandingMessages` until the expected state is reached or a generous timeout expires, and assert on the settled result. This keeps the test meaningful — it still fails if the state never becomes correct — without asserting a propagation deadline.
- Or override the pre-Act assertion for the DynamoDB implementations, since `outstandingBefore` at `:91-93` is arranging rather than asserting the behaviour under test. The assertions that matter are after `ReplayCausation`.

The bulk variant fails the same way and for the same reason.

## Scope

Found while working on #4343, which is unrelated — the failures are on `master` without it, which is how it was attributed rather than guessed at. Not filed as a flake to be re-run: the cause is identified and it will keep recurring at roughly the observed rate until the assertions change.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.