Add "withResults()" for KafkaIO.write
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
I am looking at use case where we have to wait until the Kafka Write operation is completed before we can move forward with another transform. Currently, JdbcIO support withResults() which waits for the previous transform to complete as part of Wait.on(Signal) and moves on to the next.
Similarly, it would be very beneficial to have this capability on KafkaIO (and others like PubSubIO, BigQueryIO etc).
Imported from Jira [BEAM-13298](https://issues.apache.org/jira/browse/BEAM-13298). Original Jira may contain additional context.
Reported by: rdahal.
Subtask of issue #21329
Contributor guide
Research direction
Start with the KafkaIO.write API and compare the existing JdbcIO.withResults() behavior, including how it interacts with Wait.on(Signal). Review parent issue #21329 and the imported BEAM-13298 context before defining the scope. Done should mean KafkaIO supports waiting for write completion before a downstream transform proceeds, with coverage for the new behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100