linkedin / linkedin/brooklin

TaskPrefix assignment does not follow 'system.reuseExistingDestination' flag.

Open
#972 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
968
Forks
144
PR merge metrics
No merged PRs in 30d

Description

Subject of the issue

There is a corner case where two datastreams could share a task prefix & consumer group despite expecting to output to two different topics.

Steps to reproduce

Tell us how to reproduce this issue.
Create datastream name "stream-1".
Create datastream name "dedupe-1". (dedupe-1 will now contain taskPrefix 'stream-1')
Delete "stream-1"
Recreate "stream-1" with metadata "system.reuseExistingDestination=false".

Expected behaviour

I would expect both streams to have different taskPrefixes (maybe through some uuid if system.reuseExistingDestination is present), different destination topics, and different consumer groups.

Actual behaviour

Both datastreams will still have taskPrefix "stream-1" despite the fact that with system.reuseExistingDestination=false both streams now have completely different destination topics.
The GroupIdConstructor logic will then assign the same consumer group to both streams and thus only one will actually get data.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TaskPrefix assignment and the GroupIdConstructor logic described in the issue, then reproduce the datastream creation, deletion, and recreation sequence. Done means streams using different destination topics receive distinct task prefixes and consumer groups when system.reuseExistingDestination=false.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.