dapr / dapr/components-contrib

RabbitMQ pubsub: consumers leak on broker reconnect, parking prefetchCount messages per leaked consumer

Open
#4,540 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

## Expected Behavior

After a RabbitMQ broker restart, each subscription should end up with exactly one registered AMQP consumer on its queue, and `prefetchCount` should bound a queue's unacknowledged messages to `prefetchCount × number of subscriber instances`.

## Actual Behavior

A broker restart while messages are in flight leaves **several consumers registered per subscription**, all created by the same sidecar for the same declarative subscription. Only the most recent one delivers to the app; each earlier one stays registered and holds `prefetchCount` messages unacknowledged indefinitely.

1. The broker connection drops. Every subscription sharing the connection sees `listenMessages` fail and enters the reconnect loop in `subscribeForever`.
2. `channel.Cancel(consumerTag)` is attempted, but the channel is already gone, so the deregistration never reaches the broker: `failed to cancel consumer : Exception (504) Reason: "channel/connection is not open"`.
3. One subscriber wins `reconnect()` and creates a new shared channel; the others log `stale reconnect attempt` and loop.
4. While siblings re-declare on the new shared channel, `ensureSubscription` / `channel.Consume` fail intermittently with `Exception (503) Reason: "unexpected command received"`. Each such iteration `break`s **before** reaching the cancel, then loops and calls `channel.Consume` again with a fresh consumer tag.
5. Since the tag is a new UUID per attempt (#4326), the re-registration no longer collides with the previous one, so nothing surfaces the duplicate. The earlier registration survives on a channel that is still alive, holding its prefetch allowance.

Measured with one sidecar, one declarative subscription, `prefetchCount: 100`, and an app that returns 204 promptly:

| after | consumers on queue | messages unacknowledged |
|---|---|---|
| steady state | 1 | 1 |
| 1st broker restart | 2 | 101 |
| 2nd broker restart | 4 | 301 |

Unacknowledged is exactly `(consumers - 1) × prefetchCount + in-flight`. The parked messages never move, while the live consumer keeps draining new traffic normally — so throughput looks healthy and the loss is silent.

With `ttlInSeconds` and `enableDeadLetter` set, the parked messages become permanent loss:

1. After RabbitMQ's `consumer_timeout` (default 30 min) the broker raises `precondition_failed` on the channel and requeues everything unacknowledged.
2. Every requeued message is already past its TTL, so the whole batch is dead-lettered instead of redelivered.

We hit this in a live cluster after a single broker restart: a whole batch of messages was dead-lettered rather than redelivered.

Relevant daprd output (`--log-level debug`), one queue, single subscription:

```
failed to cancel consumer webhook-worker-incoming-webhook-2f7b12e8-...: Exception (504) Reason: "channel/connection is not open"
error in subscriber for webhook-worker-incoming-webhook in ensureSubscription: Exception (504) Reason: "channel/connection is not open"
subscriber is reconnecting in 3s ...
connectionCount: current=3 reference=3
connected with connectionCount=4
setting prefetch count to 100
connectionCount: current=4 reference=3
stale reconnect attempt
error in subscriber for webhook-worker-incoming-webhook in ensureSubscription: Exception (503) Reason: "unexpected command received"
setting prefetch count to 100
registered consumer webhook-worker-incoming-webhook-8ab7c693-... for queue webhook-worker-incoming-webhook
```

Observed on daprd 1.17.7 (components-contrib v1.17.6).

## Steps to Reproduce the Problem

1. Run a single daprd with a RabbitMQ pubsub component using `prefetchCount: 100`, `enableDeadLetter: "true"`, `ttlInSeconds: "60"`, `concurrencyMode: parallel`, `requeueInFailure: "true"`, `autoAck: "false"`, `durable: "true"`, `deliveryMode: "2"`, `publisherConfirm: "true"`.
2. Declare one `dapr.io/v2alpha1` Subscription on a topic, routed to an app that returns `[]` for `GET /dapr/subscribe` and `204` for the topic route after a short sleep, so a backlog builds.
3. Publish continuously:
`while :; do curl -s -o /dev/null -XPOST -H 'Content-Type: application/json' -d '{}' http://localhost:3500/v1.0/publish//; done`
4. Restart the broker container.
5. After ~20s, inspect:
`rabbitmqctl list_queues name messages messages_unacknowledged consumers`
`rabbitmqctl list_consumers queue_name prefetch_count`
6. Repeat step 4 a few times — it fires on roughly one restart in two or three. Adding more subscriptions to the same sidecar makes it fire more readily; with three subscriptions we reached 13 consumers and 1300 unacknowledged messages on one queue.

A self-contained `docker-compose` reproduction is available if useful.

**Root cause location:**
- `components-contrib/pubsub/rabbitmq/rabbitmq.go`: in `subscribeForever`, `channel.Cancel(consumerTag)` runs only after `listenMessages` returns (v1.17.6 lines 539–577). Any iteration that `break`s earlier — `ensureSubscription` or `channel.Consume` failing — skips the cancel entirely, and the next iteration registers a new consumer under a fresh tag.
- The unique-tag change in #4326 (suggested in #4328 as "use unique consumer tags per goroutine") removed the `NOT_ALLOWED - attempt to reuse consumer tag` error that previously made a duplicate registration fail loudly. The duplicate is now silent.

**Possible fixes:**
- Track the live consumer tag per subscription and cancel it against the *current* channel before calling `channel.Consume` again.
- Cancel on every exit path from the inner loop, not only after `listenMessages` returns.
- On a successful `reconnect()`, deregister consumers left over from the previous connection generation for that queue.

## Release Note

RELEASE NOTE: **FIX** RabbitMQ pubsub leaking AMQP consumers on broker reconnect, parking `prefetchCount` messages per leaked consumer.

Contributor guide

Open the contributing guide

Research direction

Read components-contrib/pubsub/rabbitmq/rabbitmq.go, starting at subscribeForever and tracing ensureSubscription, channel.Consume, and reconnect failure paths. Run the self-contained Docker Compose reproduction and inspect rabbitmqctl list_consumers and list_queues; done means one consumer per subscription after broker restarts and unacknowledged messages remain bounded by prefetchCount × subscriber instances.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rabbitmq
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.