galaxyproject / galaxyproject/pulsar

MQ unacknowledged retries are not very multiprocess-intelligent

Open
#344 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
46
Forks
62
Avg merge
3d 9h
Merged PRs (30d)
15

Description

If enabled with `amqp_acknowledge`, the client will request acknowledgement of messages it has sent by requesting a return message on the corresponding `_ack` exchange. If it does not receive that ack within `amqp_ack_republish_time`, it will resend the message.

On the other end, a receipt is kept of messages that have been received and acted upon, so that if acknowledgements were sent but not received, retransmission does not result in duplication of work.

If you have multiple handlers serving the same Pulsar endpoints, each one of these handlers will start an ack manager that will review any unacknowledged messages, retransmit any over `amqp_ack_republish_time`, and then go to sleep for 15 seconds before doing it again. There is no locking between processes, only internally, so multiple acknowledgement managers could retransmit the same message. This is probably harmless, but not ideal.

Additionally, the acknowledgement `persistence_directory` needs to be on a shared filesystem if handlers are located on multiple hosts.

Ideally for any unique set of runner + manager, handlers would elect one of themselves as the acknowledgement manager.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the acknowledgement manager, the `amqp_ack_republish_time` retry loop, and `persistence_directory` handling referenced in the issue. Define how handlers sharing a runner and manager should elect or lock a single acknowledgement manager, then verify that retransmission is not duplicated across processes or hosts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.