Add batching properties to the ProducerConfig for Pulsar Functions
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
When deploying a Pulsar Function, a user might need to override the batching behavior of the functions producer. Currently, the `ProducerConfig` [configuration](https://pulsar.apache.org/docs/3.1.x/functions-cli/#producerconfig) only provides two options; "DEFAULT", and "KEY_BASED".
It would be worthwhile to allow users to disable batching entirely, or have more granular control of the batching behavior, in terms of [batchingMaxBytes](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#batchingMaxBytes(int)),[batchingMaxMessages](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#batchingMaxMessages(int)), and [batchingMaxPublishDelay](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#batchingMaxPublishDelay(long,java.util.concurrent.TimeUnit))
### Solution
Add the above three fields to the `ProducerConfig` definition and have them passed through to the ProducerBuilder class when the Pulsar Function is instantiated.
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start by locating the ProducerConfig definition and the Pulsar Function instantiation path that creates the ProducerBuilder. Trace how the existing batching options are applied, then verify that batchingMaxBytes, batchingMaxMessages, and batchingMaxPublishDelay are accepted and passed through, including the ability to disable batching.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100