apache / apache/pulsar

publish time non-monotonic on same partition

Open
#6,276 3 comments 1 reaction 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**
Publish time is not monotonic on the same partition.

**To Reproduce**
Steps to reproduce the behavior:
1. Publish events with a __single producer__ to a topic with many partitions, with any partition key.
2. Run a consumer on `key_shared` mode, for example:

```python
client = pulsar.Client(pulsar_url)
consumer = client.subscribe(
topic=pulsar_topic,
subscription_name="any_name_you_want",
consumer_name="any_name_you_want",
consumer_type=ConsumerType.KeyShared,
initial_position=InitialPosition.Earliest,
)

for i in itertools.count():
msg = consumer.receive()
ts = msg.publish_timestamp()
p_id = msg.message_id().partition()
lpt = latest_partition_timestamp[p_id]
assert ts >= lpt, f"latest timestamp of partition {p_id} was {lpt} and new event has timestamp {ts}"
latest_partition_timestamp[p_id] = ts
consumer.acknowledge(msg)
```
3. See assertion error

**Expected behavior**
No error should be raised since `publish_timestamp` on the same partition must be monotonically increasing.

**Desktop (please complete the following information):**
- OS: macOS 10.15.2

**Additional context**
- producer: https://github.com/apache/pulsar-client-go
- consumer: https://pypi.org/project/pulsar-client/
- partitions: 3

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the Python `pulsar-client` KeyShared consumer and the `pulsar-client-go` producer against a topic with three partitions. Trace the publish timestamp and partition handling in the Apache Pulsar repository; done means timestamps observed for each partition are monotonically increasing and the assertion no longer fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, java, python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.