[Bug] getTopicName in SinkRecord returns complete topic name for partitioned topic + empty partition index
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
2.10+
### Minimal reproduce step
In any existing sink used for partitioned topic, in the debugger stop in the write() method.
Inspect content of the record.
sink.write(record) gets record that provides API like
- Optional getPartitionIndex()
- Optional getTopicName()
Record/SinkRecord API assumes that getTopicName is a partitioned topic name ("topic") with index supplied separately
Sink created for partitioned topic gets topic names like "topic-partition-0/1/2/.." (aka complete topic name from getTopicName()) and partition index of Optional.empty()
if the sink uses the topic name as a destination (e.g. table name - for snowflake, BQ, others) it will route the data to multiple tables.
AFAICT, this never worked as intended, e.g. https://github.com/apache/pulsar/blob/a9037334a399af905fae94d2aefa5db339cbd5b1/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java#L131-L134
### What did you expect to see?
getTopicName() return partitoined topic name, getPartitionIndex() return actual partition index
### What did you see instead?
see above
### Anything else?
It is not that hard to fix but hard to estimate what it will break for existing connectors.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java at the linked lines and trace the SinkRecord getTopicName()/getPartitionIndex() contract. Check how partitioned-topic records are built and how existing connectors consume these values. Done means the topic name and actual partition index are returned separately without silently breaking connector expectations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100