apache / apache/pulsar

Let a consumer postpone or reset the acknowledgement timeout of a message

Open
#16,883 1 comment 0 reactions 0 assignees View on GitHub
Stale type/feature
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Is your feature request related to a problem? Please describe.**
I'm building a distributed job queue using a shared subscription. Job processing time range from 1 minute to 12 hours.
I'm looking for a reliable way to redeliver a message when a consumer (job worker) crash with the following requirements:
1. Avoid processing the same message several times unless a consumer crashed or unreachable for a long time.
2. Redeliver a message in a short time (10 minutes) after a consumer became unreachable without waiting for the maximum possible job processing time (12 hours).

**Describe the solution you'd like**
I'm looking for a way to let a consumer notify the broker that it is still processing a message to avoid unnecessarily redelivering it. I will set [Acknowledgement timeout](https://pulsar.apache.org/docs/2.4.0/concepts-messaging/#acknowledgement-timeout) of 10 minutes on the topic. The consumer will send a 'RESET_ACK_TIMEOUT' or 'WORKING' command for this message to the broker every 5 minutes. The broker will redeliver unacknowledged messages only after ack-timeout has passed since the last reset-ack-timeout was sent by the consumer.

**Describe alternatives you've considered**
I've considered [Negative acknowledgement](https://pulsar.apache.org/docs/2.4.0/concepts-messaging/#negative-acknowledgement), [Acknowledgement timeout](https://pulsar.apache.org/docs/2.4.0/concepts-messaging/#acknowledgement-timeout) and [Dead letter topic](https://pulsar.apache.org/docs/2.4.0/concepts-messaging/#dead-letter-topic) but couldn't find a way to reset the ackTimeout of a single message.

**Additional context**
[Disque](https://github.com/antirez/disque) is a distributed job queue by the creator of redis. You can [Add a job](https://github.com/antirez/disque#addjob-queue_name-job-ms-timeout-replicate-count-delay-sec-retry-sec-ttl-sec-maxlen-count-async) and specify a retry timeout.
With the [WORKING joibid](https://github.com/antirez/disque#working-jobid) command you can:
>Claims to be still working with the specified job, and asks Disque to postpone the next time it will deliver the job again. The next delivery is postponed for the job retry time.

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are identified in the issue. Start by reviewing Pulsar's acknowledgement-timeout, negative-acknowledgement, and dead-letter-topic behavior, then trace how the consumer and broker handle acknowledgement state. Done would require an agreed design and corresponding implementation and tests for postponing or resetting one message's timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.