confluentinc / confluentinc/confluent-kafka-javascript

Question: Consumer Configuration Recommendations for 1000+ Consumer Production System

Open
#284 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
304
Forks
45
Avg merge
11h 47m
Merged PRs (30d)
5

Description

Hi all,

I'm in the process of doing a KafkaJS -> @confluentinc/kafka-javascript migration for my company. We heavily use Kafka as our backbone, and we have > 1000 consumer groups across hundreds of topics.

Right now, we use KafkaJS with no special configuration for the consumer. When swapping out the two libraries in place, we run into OOM issues upon startup.

Are there recommendations on configuration options we should set to reach more or less equivalent behavior to KafkaJS? I've found the following options, and tried setting them a few different times, but still running into problems:

```
'fetch.max.bytes'
'queued.max.messages.kbytes'
'queued.min.messages'
'fetch.message.max.bytes'
```

What are reasonable values to consider here? Are there other parameters we should look to tweak?

Thanks in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.