BrighterCommand / BrighterCommand/Brighter
GCP Pub/Sub: Requeue(message, TimeSpan.Zero) does not redeliver immediately on real Pub/Sub
- Dominant language
- C#
- Stars
- 2.5k
- Forks
- 296
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 21
Description
## Summary
`Requeue(message, TimeSpan.Zero)` on a GCP Pub/Sub **Pull** subscription does not redeliver the message
promptly. The message *is* eventually redelivered, but only after roughly one or two ack-deadline
periods — not immediately, as `ModifyAckDeadline(ackId, 0)` is supposed to produce.
This is FR-15 of the universal transport conformance suite (#4240). It is the **only** GCP conformance
behaviour still failing, and it is failing on **every** Pull variant.
## Evidence
`gcp-ci` job [`101865520077`](https://github.com/BrighterCommand/Brighter/actions/runs/34161293474/job/101865520077),
run against **real Pub/Sub**. All four `requeuing_a_failed_message_with_zero_delay` cells fail:
| test | elapsed before redelivery |
|---|---|
| `Pull.Reactor` | `00:00:24.075` |
| `Pull.Proactor` | `00:00:19.518` |
| `PullOrdering.Reactor` | `00:00:11.308` |
| `PullOrdering.Proactor` | `00:00:22.058` |
```
Expected redelivery within 5 s of Requeue(M, TimeSpan.Zero); elapsed: 00:00:24.0750433
at ...Pull.Reactor.WhenRequeuingAFailedMessageWithZeroDelayShouldRedeliverImmediately
...MessagingGateway/Pull/Generated/Reactor/When_requeuing_a_failed_message_with_zero_delay_should_redeliver_immediately.cs:line 79
```
**The subscription's ack deadline is 10 s** (`tests/Paramore.Brighter.Gcp.Tests/MessagingGateway/GcpPullMessageGatewayProvider.cs:138`).
The four elapsed times are ≈1× and ≈2× that deadline. That is the signature of **redelivery arriving on
ack-deadline expiry rather than on the modack** — i.e. the `ModifyAckDeadline(..., 0)` appears not to be
taking effect.
## This is not flake
Measured twice, on two different branch heads, with 4/4 failing both times. In the same period every
neighbouring GCP failure went green — so this is not general instability:
| behaviour | FR | 21 → 10 → **5** failures |
|---|---|---|
| `requeuing_with_zero_delay` | **FR-15** | 4 → 4 → **4 — never moved** |
| `nacking_a_message` | FR-16 | 4 → 1 → **1** |
| `sending_a_delayed_message` | FR-9 | 3 → 2 → **0** |
| `nacking_first_of_two` | FR-16 | 4 → 2 → **0** |
| `requeuing_should_be_redelivered` | FR-22 | 4 → 1 → **0** |
Note especially that FR-15 was **untouched by the ack-deadline fix** (`6c1fd91d2`), which took the other
four rows down. That fix shortened `ackDeadlineSeconds` to 10 so that *nack*-driven redelivery (a no-op
in this gateway, so it waits for the deadline) could land inside the tests' 30 s ceiling. FR-15 does not
depend on the deadline — it is supposed to bypass it — so the fix could not help it, and did not.
## The code
`src/Paramore.Brighter.MessagingGateway.GcpPubSub/GcpPullMessageConsumer.cs:309-332` (and the async twin
at `:343-372`) look correct on inspection:
```csharp
// The requeue policy is defined by subscription, during its creation
client.ModifyAckDeadline(subscriptionName, [ackId], 0);
```
**Root cause is not established.** Leads worth checking, in rough order of promise:
1. **Is the `ackId` still valid at the point of the modack?** If the pump's lease management has already
extended or invalidated it, the modack would be accepted-but-ineffective and redelivery would fall
back to the deadline — which is exactly the timing observed.
2. **`RetryPolicy` is null here**, so exponential backoff is *not* the explanation: `Subscription`
defaults `RequeueDelay` to `TimeSpan.Zero` (`src/Paramore.Brighter/Subscription.cs:229`) and
`GcpPubSubMessageGateway` only sets a `RetryPolicy` when it is non-zero (`:369-376`, `:449-461`).
Worth confirming against the live subscription rather than trusting the read.
3. The doc comments claim redelivery happens "according to the subscription's retry policy" — with no
retry policy configured, it is worth establishing what Pub/Sub actually guarantees for
modack-to-zero, and whether "immediately" is a promise the gateway can make at all.
If it turns out Pub/Sub will not honour modack-to-zero as an immediate redelivery, then FR-15 is a
**platform difference** rather than a defect and the ledger cell should say so — but that needs
establishing, not assuming.
## ⚠️ The green GCP cells rest on the emulator
The certification commit `d94e9b827` says so in its own message: *"GCP / Pull resolved against a local
Pub/Sub emulator … No real-GCP creds available locally; CI runs GCP against real Pub/Sub."* The emulator
certified these cells green; real Pub/Sub does not agree. **Treat every green GCP cell as unproven
against real Pub/Sub until CI says otherwise** — this one is simply the first to be caught.
Contributor guide
Assessment
This issue has not been assessed yet.