Schema prevents usage of Dead Letter Queue DLQ
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 142
Description
**Describe the bug**
When using a Schema pushing a negativeAcknowledged Message to a DeadletterQueue will fail.
There is no obvious Errors/Exceptions. The Message, just stay in the current Topic and is never published to the DLQ-Topic.
When using no Schema oder Schema.STRING it will work.
This happende when using pulsar 2.7.1 and the Java-Client.
Pulsar was started as standalone docker container.
```
$ docker run -it \
-p 6650:6650 \
-p 8080:8080 \
--mount source=pulsardata,target=/pulsar/data \
--mount source=pulsarconf,target=/pulsar/conf \
apachepulsar/pulsar:2.7.1 \
bin/pulsar standalone
```
**To Reproduce**
I created an repo with some code for this issue: https://github.com/jvoigt/pulsar-schema-dlq-issue
Steps to reproduce the behavior:
1. Create Producer to topic with Schema `pulsarClient.newProducer(JSONSchema.of(Payload.class)).create()`
2. Send Message to Producer `producer.send( new Payload("My Json message"))`
3. Consume Topic with Schema AND DLQ `pulsarClient.newConsumer(schema).topic(topic) .deadLetterPolicy(DeadLetterPolicy.builder().maxRedeliverCount(3).build()).subscribe()`
4. receive the Message, but use a nack. `consumer.negativeAcknowledge(msg);`
5. The Message will be redeliverd 3 Times, then the DLQ-Topic ist automatically create.
6. But the Message is never sent.
**Logs**
See the attached Repo for detailed Logs:
But Mainly you can see that the client is creating a new Producer for the DLQ-Topic and then nothing more happens.
```
Apr 26, 2021 8:19:13 AM com.jvoigt.issue.MyListener received
INFO: persistent://public/default/jsonTopic => Message received: {"message":"My Json message"}
Apr 26, 2021 8:19:13 AM com.jvoigt.issue.MyListener received
SEVERE: persistent://public/default/jsonTopic => I WILL NACK IT with Counter: 3
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.UnAckedMessageTracker$2 run
WARNING: [ConsumerBase{subscription='subscription', consumerName='5416a', topic='jsonTopic'}] 1 messages have timed-out
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Starting Pulsar producer perf with config: {
"topicName" : "jsonTopic-subscription-DLQ",
.....
}
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Pulsar client config: {
"serviceUrl" : "pulsar://localhost:6650",
.....
}
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerImpl connectionOpened
INFO: [jsonTopic-subscription-DLQ] [null] Creating producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerImpl lambda$connectionOpened$16
INFO: [jsonTopic-subscription-DLQ] [standalone-6-1] Created producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]
Apr 26, 2021 8:20:04 AM org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl lambda$init$0
INFO: [jsonTopic] [subscription] [5416a] Prefetched messages: 0 --- Consume throughput received: 0.07 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 0.00 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Starting Pulsar producer perf with config: {
"topicName" : "jsonTopic-subscription-DLQ",
.....
}
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Pulsar client config: {
"serviceUrl" : "pulsar://localhost:6650",
.....
}
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerImpl connectionOpened
INFO: [jsonTopic-subscription-DLQ] [null] Creating producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerImpl lambda$connectionOpened$16
INFO: [jsonTopic-subscription-DLQ] [standalone-6-2] Created producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]`
```
thank you all in advance :)
Contributor guide
Research direction
Start with the linked reproduction repository and the Java client flow using JSONSchema, negativeAcknowledge, and DeadLetterPolicy. Trace the dead-letter producer behavior after the third redelivery and verify the result with the standalone Pulsar 2.7.1 setup. Done means a schema-backed message is published to the automatically created DLQ topic instead of remaining in the current topic without an error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100