apache / apache/pulsar

Add batching properties to the ProducerConfig for Pulsar Functions

Open
#21,758 2 comments 1 reaction 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

When deploying a Pulsar Function, a user might need to override the batching behavior of the functions producer. Currently, the `ProducerConfig` [configuration](https://pulsar.apache.org/docs/3.1.x/functions-cli/#producerconfig) only provides two options; "DEFAULT", and "KEY_BASED".

It would be worthwhile to allow users to disable batching entirely, or have more granular control of the batching behavior, in terms of [batchingMaxBytes](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#batchingMaxBytes(int)),[batchingMaxMessages](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#batchingMaxMessages(int)), and [batchingMaxPublishDelay](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#batchingMaxPublishDelay(long,java.util.concurrent.TimeUnit))

### Solution

Add the above three fields to the `ProducerConfig` definition and have them passed through to the ProducerBuilder class when the Pulsar Function is instantiated.

### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by locating the ProducerConfig definition and the Pulsar Function instantiation path that creates the ProducerBuilder. Trace how the existing batching options are applied, then verify that batchingMaxBytes, batchingMaxMessages, and batchingMaxPublishDelay are accepted and passed through, including the ability to disable batching.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.