apache / apache/pulsar

Schema prevents usage of Dead Letter Queue DLQ

Open
#10,377 7 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**
When using a Schema pushing a negativeAcknowledged Message to a DeadletterQueue will fail.
There is no obvious Errors/Exceptions. The Message, just stay in the current Topic and is never published to the DLQ-Topic.
When using no Schema oder Schema.STRING it will work.

This happende when using pulsar 2.7.1 and the Java-Client.

Pulsar was started as standalone docker container.
```
$ docker run -it \
-p 6650:6650 \
-p 8080:8080 \
--mount source=pulsardata,target=/pulsar/data \
--mount source=pulsarconf,target=/pulsar/conf \
apachepulsar/pulsar:2.7.1 \
bin/pulsar standalone
```

**To Reproduce**
I created an repo with some code for this issue: https://github.com/jvoigt/pulsar-schema-dlq-issue

Steps to reproduce the behavior:
1. Create Producer to topic with Schema `pulsarClient.newProducer(JSONSchema.of(Payload.class)).create()`
2. Send Message to Producer `producer.send( new Payload("My Json message"))`
3. Consume Topic with Schema AND DLQ `pulsarClient.newConsumer(schema).topic(topic) .deadLetterPolicy(DeadLetterPolicy.builder().maxRedeliverCount(3).build()).subscribe()`
4. receive the Message, but use a nack. `consumer.negativeAcknowledge(msg);`
5. The Message will be redeliverd 3 Times, then the DLQ-Topic ist automatically create.
6. But the Message is never sent.

**Logs**
See the attached Repo for detailed Logs:
But Mainly you can see that the client is creating a new Producer for the DLQ-Topic and then nothing more happens.

```
Apr 26, 2021 8:19:13 AM com.jvoigt.issue.MyListener received
INFO: persistent://public/default/jsonTopic => Message received: {"message":"My Json message"}
Apr 26, 2021 8:19:13 AM com.jvoigt.issue.MyListener received
SEVERE: persistent://public/default/jsonTopic => I WILL NACK IT with Counter: 3
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.UnAckedMessageTracker$2 run
WARNING: [ConsumerBase{subscription='subscription', consumerName='5416a', topic='jsonTopic'}] 1 messages have timed-out
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Starting Pulsar producer perf with config: {
"topicName" : "jsonTopic-subscription-DLQ",
.....
}
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Pulsar client config: {
"serviceUrl" : "pulsar://localhost:6650",
.....
}
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerImpl connectionOpened
INFO: [jsonTopic-subscription-DLQ] [null] Creating producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]
Apr 26, 2021 8:19:16 AM org.apache.pulsar.client.impl.ProducerImpl lambda$connectionOpened$16
INFO: [jsonTopic-subscription-DLQ] [standalone-6-1] Created producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]
Apr 26, 2021 8:20:04 AM org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl lambda$init$0
INFO: [jsonTopic] [subscription] [5416a] Prefetched messages: 0 --- Consume throughput received: 0.07 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 0.00 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Starting Pulsar producer perf with config: {
"topicName" : "jsonTopic-subscription-DLQ",
.....
}
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerStatsRecorderImpl init
INFO: Pulsar client config: {
"serviceUrl" : "pulsar://localhost:6650",
.....
}
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerImpl connectionOpened
INFO: [jsonTopic-subscription-DLQ] [null] Creating producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]
Apr 26, 2021 8:20:24 AM org.apache.pulsar.client.impl.ProducerImpl lambda$connectionOpened$16
INFO: [jsonTopic-subscription-DLQ] [standalone-6-2] Created producer on cnx [id: 0x7861b6d7, L:/127.0.0.1:43226 - R:localhost/127.0.0.1:6650]`
```

thank you all in advance :)

Contributor guide

Open the contributing guide

Research direction

Start with the linked reproduction repository and the Java client flow using JSONSchema, negativeAcknowledge, and DeadLetterPolicy. Trace the dead-letter producer behavior after the third redelivery and verify the result with the standalone Pulsar 2.7.1 setup. Done means a schema-backed message is published to the automatically created DLQ topic instead of remaining in the current topic without an error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.