BrighterCommand / BrighterCommand/Brighter
SQS: requeueCount is inert — requeue does not persist handled-count, so ADR 0038's direct DLQ send is unreachable on budget exhaustion
- Dominant language
- C#
- Stars
- 2.5k
- Forks
- 296
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 21
Description
## Summary
On SQS, `requeueCount` is accepted, configured, and has **no effect**. A subscription's delivery
budget can never be exhausted, so the pump never rejects on budget exhaustion, so the direct DLQ
send that [ADR 0038](../blob/master/docs/adr/0038-aws-sqs-dlq-direct-send.md) specifies is
unreachable for that case. Messages are redelivered without limit until the queue's native
`maxReceiveCount` redrive rescues them.
This is **not** a proposal to change the DLQ strategy. ADR 0038 already settled that, and the path it
specifies works. The gap is one step upstream, in requeue, and it makes part of ADR 0038 dead code.
## Why this is a gap and not the intended design
ADR 0038 ("AWS SQS DLQ: Replace ChangeMessageVisibility with Direct Send", Accepted 2026-02-11)
decided that when `DeadLetterRoutingKey` is configured, `Reject` sends directly to the Brighter DLQ
and deletes the original. It explicitly considered and rejected the alternative of leaning on redrive:
> **Keep ChangeMessageVisibility(0) and Add Brighter DLQ Alongside** … **Rejected**: The visibility
> trick is always inferior — it causes re-processing cycles.
and its coexistence table records: Brighter DLQ configured + any native redrive policy →
*"Brighter sends directly to its configured DLQ queue. Native policy is irrelevant for rejected
messages."* Direct send does not fight redrive, because the reject path **deletes** the source
message, so redrive's counter never comes into play for it.
That path is implemented and healthy: `RejectAsync` routes through `_deadLetterProducer`
(`SqsMessageConsumer.cs:281-294`), and conformance behaviour FR-4 (explicit `Reject` → DLQ) is `Pass`
on all eight AWS / AWS.V4 configurations.
What cannot happen is reaching it by spending the delivery budget.
## Root cause
The budget is enforced by the pump, which increments the count and then tests it
(`Reactor.cs:494-509`, `Proactor.cs:500-513`):
```
message.Header.UpdateHandledCount();
if (message.HandledCountReached(RequeueCount)) -> RejectMessage(..., DeliveryError)
```
On SQS the count never gets above 1:
1. `SqsMessageConsumer.RequeueAsync` requeues by calling `ChangeMessageVisibilityAsync`
(`SqsMessageConsumer.cs:384-405`). The stored SQS message is made visible again and is **never
rewritten**.
2. `handled-count` is written only when a message is *sent* — `SqsMessageSender.cs:130`,
`SnsMessagePublisher.cs:110`. A visibility change is not a send.
3. So every redelivery is parsed from the unchanged stored copy and arrives with `handled-count` at
its original value (`SqsMessageCreator.cs:323`, `SqsInlineMessageCreator.cs:350`). The pump
increments it to 1 in memory, `HandledCountReached(3)` is false, and it requeues again — forever.
Both packages are affected identically, as ADR 0038's lockstep requirement anticipates:
`Paramore.Brighter.MessagingGateway.AWSSQS` and `…AWSSQS.V4`
(V4: `SqsMessageConsumer.cs:346,394`; `SqsMessageSender.cs:128`; `SnsMessagePublisher.cs:110`).
## How it was measured
Found while running conformance behaviour FR-23 (*requeue budget exhausted to DLQ*) from
[#4297](https://github.com/BrighterCommand/Brighter/pull/4297) against LocalStack, with a real
`Reactor`/`Proactor` driving the channel and a handler that always defers. Subscription configured
with `requeueCount: 3` and a redrive policy.
The message does reach the DLQ, but it arrives as:
```
handledCount=0 id= (bag: handled-count=0, ReceiptHandle=…)
```
with **no rejection metadata**. `RefreshMetadata` stamps that metadata inside `RejectAsync`, so its
absence is direct evidence that Brighter never rejected the message — SQS redrive moved its own
stored copy.
Raising the harness's `maxReceiveCount` above `requeueCount`, to stop the broker answering first, was
also tried. It changes nothing, and could not: with the count resetting on every delivery there is no
budget to exhaust, so the only effect is that redrive takes longer to fire.
## Suggested direction
The cooperative fix needs no new counter and does not work against redrive: **use SQS's own.**
`SqsMessageConsumer` already requests `MessageSystemAttributeNames = ["All"]` on every receive
(`SqsMessageConsumer.cs:188-194`; V4 `:275-276`), so `ApproximateReceiveCount` — SQS's own delivery
counter — is already on the wire on every message. Nothing in `src/` reads it; `grep -rn
"ApproximateReceiveCount" src` returns nothing at all. Mapping it into `HandledCount` in the message
creators would make `requeueCount` work off the broker's counter, which is exactly the number SQS
itself redrives on.
Alternatives, for completeness:
- Make requeue re-send the message rather than change visibility, as RMQ does, so headers persist.
Heavier, and it changes delivery semantics (new receipt handle, FIFO ordering implications).
- Document `requeueCount` as unsupported on SQS and have the subscription reject or warn when it is
set alongside a `DeadLetterRoutingKey`, rather than accepting it silently.
Whichever way it goes, one consequence is worth documenting explicitly: when both are configured, the
effective delivery limit is `min(requeueCount, maxReceiveCount)`, and today that is always
`maxReceiveCount`.
## Impact
- `requeueCount` is silently inert on SQS/SNS — configured, accepted, ignored.
- Users get unbounded redelivery bounded only by `maxReceiveCount`, which may be unset.
- Messages that do reach the DLQ by redrive carry **no rejection metadata**, so ADR 0036's
enrichment (`RejectionReason`, `RejectionMessage`, `OriginalTopic`, `RejectionTimestamp`) is absent
for this route — DLQ consumers cannot tell why the message is there.
- The eight `AWS` / `AWS.V4` cells for FR-23 are held `Deferred` in
`specs/0036-universal-transport-conformance-tests/conformance-status.md` pending this decision.
Contributor guide
Assessment
This issue has not been assessed yet.