Handle failures sending messages to RabbitMQ
- Dominant language
- TypeScript
- Stars
- 88
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
We have had a couple incidents where the API just stops sending messages to
RabbitMQ and we don't even know about it, since the library we use abstracts
the underlying connection and failures. I believe it just accumulates messages in
memory and keeps trying to reconnect forever, hoping it will eventually succeed.
There is a way to send a callback to the publish function to make sure we only return
a success to the users if we do have a success publishing the message though:
https://www.npmjs.com/package/amqp-connection-manager#channelwrapperpublish-and-channelwrappersendtoqueue
Update: This is even weirder now, from the docs, since they claim that if a callback
is not sent, the returned promise will only be fulfilled when the publish actually happens.
So this needs further investigation. My immediate suspicion is that the lib is not actually
checking broker confirmations (like a reverse ACK) and we need to enable that somehow.
Contributor guide
Research direction
Start by tracing the API's publish calls and reviewing amqp-connection-manager's ChannelWrapper.publish and ChannelWrapper.sendToQueue behavior, especially broker confirmations and callback handling. Determine how failed publishes are currently reported; done means confirmed failures are surfaced instead of being silently retried or buffered, with the relevant behavior verified by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rabbitmq, typescript
- Domain
- api, backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100