galaxyproject / galaxyproject/pulsar
MQ unacknowledged retries are not very multiprocess-intelligent
- Dominant language
- Python
- Stars
- 46
- Forks
- 62
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 15
Description
If enabled with `amqp_acknowledge`, the client will request acknowledgement of messages it has sent by requesting a return message on the corresponding `_ack` exchange. If it does not receive that ack within `amqp_ack_republish_time`, it will resend the message.
On the other end, a receipt is kept of messages that have been received and acted upon, so that if acknowledgements were sent but not received, retransmission does not result in duplication of work.
If you have multiple handlers serving the same Pulsar endpoints, each one of these handlers will start an ack manager that will review any unacknowledged messages, retransmit any over `amqp_ack_republish_time`, and then go to sleep for 15 seconds before doing it again. There is no locking between processes, only internally, so multiple acknowledgement managers could retransmit the same message. This is probably harmless, but not ideal.
Additionally, the acknowledgement `persistence_directory` needs to be on a shared filesystem if handlers are located on multiple hosts.
Ideally for any unique set of runner + manager, handlers would elect one of themselves as the acknowledgement manager.
Contributor guide
Research direction
Start by tracing the acknowledgement manager, the `amqp_ack_republish_time` retry loop, and `persistence_directory` handling referenced in the issue. Define how handlers sharing a runner and manager should elect or lock a single acknowledgement manager, then verify that retransmission is not duplicated across processes or hosts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100