apache / apache/pulsar

[Bug] getTopicName in SinkRecord returns complete topic name for partitioned topic + empty partition index

Open
#19,922 3 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

2.10+

### Minimal reproduce step

In any existing sink used for partitioned topic, in the debugger stop in the write() method.
Inspect content of the record.

sink.write(record) gets record that provides API like

- Optional getPartitionIndex()
- Optional getTopicName()

Record/SinkRecord API assumes that getTopicName is a partitioned topic name ("topic") with index supplied separately

Sink created for partitioned topic gets topic names like "topic-partition-0/1/2/.." (aka complete topic name from getTopicName()) and partition index of Optional.empty()
if the sink uses the topic name as a destination (e.g. table name - for snowflake, BQ, others) it will route the data to multiple tables.

AFAICT, this never worked as intended, e.g. https://github.com/apache/pulsar/blob/a9037334a399af905fae94d2aefa5db339cbd5b1/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java#L131-L134

### What did you expect to see?

getTopicName() return partitoined topic name, getPartitionIndex() return actual partition index

### What did you see instead?

see above

### Anything else?

It is not that hard to fix but hard to estimate what it will break for existing connectors.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java at the linked lines and trace the SinkRecord getTopicName()/getPartitionIndex() contract. Check how partitioned-topic records are built and how existing connectors consume these values. Done means the topic name and actual partition index are returned separately without silently breaking connector expectations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.