publish time non-monotonic on same partition
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 142
Description
**Describe the bug**
Publish time is not monotonic on the same partition.
**To Reproduce**
Steps to reproduce the behavior:
1. Publish events with a __single producer__ to a topic with many partitions, with any partition key.
2. Run a consumer on `key_shared` mode, for example:
```python
client = pulsar.Client(pulsar_url)
consumer = client.subscribe(
topic=pulsar_topic,
subscription_name="any_name_you_want",
consumer_name="any_name_you_want",
consumer_type=ConsumerType.KeyShared,
initial_position=InitialPosition.Earliest,
)
for i in itertools.count():
msg = consumer.receive()
ts = msg.publish_timestamp()
p_id = msg.message_id().partition()
lpt = latest_partition_timestamp[p_id]
assert ts >= lpt, f"latest timestamp of partition {p_id} was {lpt} and new event has timestamp {ts}"
latest_partition_timestamp[p_id] = ts
consumer.acknowledge(msg)
```
3. See assertion error
**Expected behavior**
No error should be raised since `publish_timestamp` on the same partition must be monotonically increasing.
**Desktop (please complete the following information):**
- OS: macOS 10.15.2
**Additional context**
- producer: https://github.com/apache/pulsar-client-go
- consumer: https://pypi.org/project/pulsar-client/
- partitions: 3
Contributor guide
Research direction
Start by reproducing the issue with the Python `pulsar-client` KeyShared consumer and the `pulsar-client-go` producer against a topic with three partitions. Trace the publish timestamp and partition handling in the Apache Pulsar repository; done means timestamps observed for each partition are monotonically increasing and the assertion no longer fails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, java, python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100