apache / apache/beam

Allow Inheritance from KafkaUnboundedSource

Open
#21,047 0 comments 0 reactions 0 assignees View on GitHub
io java kafka new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Reason:

We have dataflow to call one internal API (the API call take 60ms). So we want to start more threads to call this API. Kafka IO thread is control by split by default. And split is control by total partitions and system suggestion number. And system suggestion number is 4 * max workers. which is too low for us.  

Solution:

Use facture to allow user inheritance KafkaUnboundedSource.  and override the split function like

 
```

// code placeholder
@Override
public List> split(int desiredNumSplits,
PipelineOptions options)
throws Exception {
int split = this.getMaxSplits();

if(split <= 0){
return super.split(desiredNumSplits, options);
}else{

return super.split(split, options);
}
}

```

 this may also useful for Override other feature of the KafkaUnboundedSource without copy all the io code.

Imported from Jira [BEAM-12631](https://issues.apache.org/jira/browse/BEAM-12631). Original Jira may contain additional context.
Reported by: renhongxiang.

Contributor guide

Open the contributing guide

Research direction

Start by reading KafkaUnboundedSource and its split method to understand which inheritance or visibility change is needed. Done means users can subclass KafkaUnboundedSource and override split without copying the Kafka IO implementation, while preserving the existing default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.