confluentinc / confluentinc/confluent-kafka-javascript

RangeError: Too many elements passed to Promise.all

Open
#273 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
304
Forks
45
Avg merge
11h 47m
Merged PRs (30d)
5

Description

**Environment Information**
- OS [e.g. Mac, Arch, Windows 10]: Node JS Alpine container
- Node Version [e.g. 8.2.1]: v18.20.5
- NPM Version [e.g. 5.4.2]: 10.8.2
- C++ Toolchain [e.g. Visual Studio, llvm, g++]:
- confluent-kafka-javascript version [e.g. 2.3.3]: 1.2.0

**Steps to Reproduce**
use many `producer.send` calls, each with a big array of messages

**confluent-kafka-javascript Configuration Settings**
```
import { KafkaJS } from "@confluentinc/kafka-javascript";

const Kafka = KafkaJS.Kafka;
const kafakStr = "ws";

const kafka = new Kafka({
kafkaJS: {
clientId: kafakStr,
brokers: [process.env.KAFKA],
},
});

const kafkaObj = {
producer: kafka.producer({
"batch.size": 100000,
"linger.ms": 50,
"compression.type": "lz4",
kafkaJS: { acks: 1 },
}),
consumer: kafka.consumer({ kafkaJS: { groupId: kafakStr } }),
};
```

**Additional context**
my app is connected in a websocket to stock market data provider.
the provider is sending messages in a very fast rate, as much as 500,000 messages per 10s.
to avoid WS delays and messages drop, I move the load to later stages, by simply sending every incoming message in kafka.
the kafka producer is configure with "high throughput" parameters (see below).
the problematic time is around the market close (16:15 US ET), that's when the rate is fastest
the first symptom I saw was that the Node JS process was getting out of memory.
then I used `--max-old-space-size=8192` in Node JS, and tried to avoid too many promises spawning, so instead of sending each message in a different promise, I aggregate them into 2 interchanging buckets, that are being sent every 10 ms.
after this latest change, I started getting the `RangeError: Too many elements passed to Promise.all` error.
it seems that still, with or without aggregating the messages (in the app, before sending) every message has a promise created for it.

any help is appreciated!

logs:
```
2025-03-12 16:13:47.349 Memory usage: RSS=598.38MB | HeapUsed=209.62MB | arrayBuffers=10.77MB | external=12.12MB | heapTotal=350.04MB
2025-03-12 16:14:17.925 Memory usage: RSS=1174.94MB | HeapUsed=538.16MB | arrayBuffers=57.72MB | external=59.06MB | heapTotal=726.14MB
2025-03-12 16:14:38.151 Memory usage: RSS=1781.49MB | HeapUsed=1254.74MB | arrayBuffers=82.97MB | external=84.31MB | heapTotal=1298.77MB
2025-03-12 16:14:58.199 Memory usage: RSS=2210.79MB | HeapUsed=1687.32MB | arrayBuffers=71.02MB | external=72.37MB | heapTotal=1743.24MB
2025-03-12 16:15:18.199 Memory usage: RSS=3028.63MB | HeapUsed=2464.37MB | arrayBuffers=91.61MB | external=92.95MB | heapTotal=2531.88MB
2025-03-12 16:15:39.413 Memory usage: RSS=3719.82MB | HeapUsed=3098.89MB | arrayBuffers=122.87MB | external=124.21MB | heapTotal=3177.13MB
2025-03-12 16:16:45.464 !!!!!!!!!!! uncaught exception: RangeError: Too many elements passed to Promise.all
2025-03-12 16:16:45.464 RangeError: Too many elements passed to Promise.all
at Function.all ()
at Producer.send (/app/node_modules/.pnpm/@confluentinc+kafka-javascript@1.2.0/node_modules/@confluentinc/kafka-javascript/lib/kafkajs/_producer.js:654:45)
at Timeout._onTimeout (file:///app/server.js:198:22)
at listOnTimeout (node:internal/timers:569:17)
at process.processTimers (node:internal/timers:512:7)
```

code:
```
setInterval(() => {
if (busySending) return;
busySending = true;

if (arrFilling === "A") {
arrFilling = "B";
if (messagesA.length) {
kafka.producer.send({ topic: "trades", messages: messagesA }).then(() => {
messagesA = [];
busySending = false;
});
} else {
busySending = false;
}
} else {
arrFilling = "A";
if (messagesB.length) {
kafka.producer.send({ topic: "trades", messages: messagesB }).then(() => {
messagesB = [];
busySending = false;
});
} else {
busySending = false;
}
}
}, 10);
```

Contributor guide

Open the contributing guide

Research direction

Start at lib/kafkajs/_producer.js:654, where Producer.send reaches Promise.all, and inspect the call from server.js:198. Reproduce the issue with many producer.send calls and large message arrays under the reported Node.js and Kafka configuration. Done means the reported workload no longer raises RangeError: Too many elements passed to Promise.all.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, kafka, node.js
Domain
backend, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.