confluentinc / confluentinc/confluent-kafka-javascript

kafkaJS consumer: maxBytesPerPartition is silently overridden by maxBytes default

Open Beginner friendly
#511 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
304
Forks
45
Avg merge
11h 47m
Merged PRs (30d)
5

Description

librdkafka [`CONFIGURATION.md`](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) documents `max.partition.fetch.bytes` as an alias of `fetch.message.max.bytes` (same underlying librdkafka property). The kafkaJS compat layer in `lib/kafkajs/_consumer.js` always writes both keys when building the librdkafka config, regardless of which one the caller actually set:

```js
// lib/kafkajs/_consumer.js:576-579
if (Object.hasOwn(kjsConfig, 'maxBytesPerPartition')) {
rdKafkaConfig['max.partition.fetch.bytes'] = kjsConfig.maxBytesPerPartition;
} else {
rdKafkaConfig['max.partition.fetch.bytes'] = 1048576; // 1 MB
}
...
// lib/kafkajs/_consumer.js:590-593
if (Object.hasOwn(kjsConfig, 'maxBytes')) {
rdKafkaConfig['fetch.message.max.bytes'] = kjsConfig.maxBytes;
} else {
rdKafkaConfig['fetch.message.max.bytes'] = 10485760; // 10 MB
}
```

Because both statements assign the same underlying librdkafka property (aliases), and the `fetch.message.max.bytes` assignment runs second, it wins. A caller who sets only `maxBytesPerPartition` (intending to bound per-partition fetch size) gets silently overridden by the unset `maxBytes` default of 10 MB — the per-partition bound never takes effect.

**Environment Information**
- OS: Ubuntu 24.04.4 LTS (WSL2)
- Node Version: v24.16.0
- NPM Version: 11.13.0
- C++ Toolchain: not applicable (repro uses the published, prebuilt binary)
- confluent-kafka-javascript version: 1.10.0

**Steps to Reproduce**

A minimal standalone repro (docker-compose Kafka broker + Node script) is available here: https://github.com/rafaelgpimenta/repro-maxbytes-bug.

1. Construct a consumer with `maxBytesPerPartition: 65536` and do not set `maxBytes`.
2. Enable `debug: 'conf'` and call `consumer.connect()`.
3. Observe `fetch.message.max.bytes` (and therefore the effective per-partition fetch bound) resolves to `10485760`, not `65536`.

**confluent-kafka-javascript Configuration Settings**

```js
const consumer = kafka.consumer({
kafkaJS: {
groupId: 'my-group',
maxBytesPerPartition: 65536, // maxBytes intentionally left unset
},
debug: 'conf',
});
```

**Additional context**

**Expected behavior**

Setting only one of the two alias keys should apply that bound; the other key's default should not silently override an explicitly configured value on the same underlying property. At minimum, the two writes should be resolved together (e.g. only default a key if the other alias is also unset), rather than always defaulting the second one independently of what the first one resolved to.

**Impact**

For consumers that only set `maxBytesPerPartition` to bound per-partition fetch size, the resulting behavior is an unbounded 10 MB per-partition fetch — a memory/correctness risk with no error or warning.

**Workaround**

Set both `maxBytesPerPartition` and `maxBytes` to the same value so the property resolves consistently regardless of write order.

Contributor guide

Open the contributing guide

Research direction

Start in lib/kafkajs/_consumer.js at lines 576-579 and 590-593, then use the standalone repro linked in the issue with debug: 'conf'. Done means setting only maxBytesPerPartition or only maxBytes applies that value without the other alias default overriding it, while the existing workaround case remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.