redpanda-data / redpanda-data/connect
Update batching mechanisms to allow upper bounded policy
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
Ran into this while testing some of our scenarios. We push a lot of data of various sizes up to 100k through kafka. The data comes in "chunks" which we split into smaller pieces which are then batched up and send via http_client. The HTTP endpoint we are sending data to has a maximum payload limit ~500k. We need to make sure that we are efficient (less round trips, so we try to send as much data as possible) so we try to batch up as much as we can without going over the limit.
Using benthos it makes it a lot more challenging. Looking at this code:
https://github.com/benthosdev/benthos/blob/main/internal/component/output/batcher/batcher.go#L109
It looks like it should be flushing multiple times for a single batch, but all it does is set a flag to flush at the end of the transaction. This is problematic for us since we keep hitting our payload limit(s).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in internal/component/output/batcher/batcher.go around line 109 and trace how batching and flush decisions reach http_client. Reproduce the Kafka scenario with chunks up to 100k and an endpoint limit of about 500k. Done means a batch can be split into multiple sends while maximizing payload size without exceeding the endpoint limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100