xmidt-org / xmidt-org/splitter
add transaction support for publishing
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 0
- Forks
- 0
- Avg merge
- 1m
- Merged PRs (30d)
- 17
Description
if we want to re-use the functionality for topic splitting in wrpkafka, then we need to add the ability to use transactions to wrpkafka too (separate story on wrpkafka) This would involve passing the franz-go client into the publisher as a session, along with a block of wrpkafka records, rather than using the configured publisher in wrpkafka and one record at a time.
This would allow for all or nothing commits, rather than potentially committing records we really want to keep trying.
We want to probably support both sync and async transactions (Produce and ProduceSync under the covers)
The caller, this repository, would then decide based on all the errors it gets back whether to mark the records in this transaction as committed or not. If the errors in the transaction are due to buffer full, network timeout etc... in other words, the network to the kafka end user topic is down, the records in the transaction would not be committed. Otherwise they would.
Note that this could still result in duplicate records because the consumer would potentially be reprocessing records that were already published. However, what we really want to accomplish with this feature is to not mark any records for commit if the network is down.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the publisher and wrpkafka integration described in the issue, including how the franz-go client is passed and how records are currently produced one at a time. Review the separate wrpkafka transaction story and the Produce and ProduceSync paths. Done means supporting transaction-based publishing while allowing the caller to decide whether records are committed from the returned errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100