grpc / grpc/grpc-node

Critical performance bottleneck with high-throughput GRPC stream handling

Open
#2,669 18 comments 1 reaction 0 assignees View on GitHub
package: @grpc/grpc-js
Dominant language
TypeScript
Stars
4.8k
Forks
716
Avg merge
2d 3h
Merged PRs (30d)
10

Description

### Problem
Our application uses this module with a Node.js backend to process incoming events from a gRPC stream. The volume of these events is substantial, frequently reaching several thousands per second, and can scale to 10-15k events per second during peaks.

Each event contains a timestamp indicating its original compilation time by the gRPC server. We've identified a significant performance limitation, this module struggles to process over ~250 events per second. Logs comparing the current time against the event timestamp lead to the processing of significantly outdated events, sometimes lagging behind by several minutes. This level of performance makes consuming a high-volume stream through a single Node.js process impractical.

### Reproduction steps
1. Access a high-throughput GRPC stream.
2. Attempt to process this high-volume stream.
3. Observe significant delays in event processing, with events not being processed timely.

Code used to test throughput:
```ts
const client = new XClient("endpoint", ChannelCredentials.createInsecure())
const stream = client.SubscribeXUpdates(new SubscribeXUpdatesRequest())

let start: number | undefined
let n = 0

stream.on("data", () => {
if (!start) start = Date.now()
n++
})

stream.on("end", () => {
console.error("Stream ended")
process.exit(1)
})

stream.on("error", (error) => {
console.error("Stream error", error)
process.exit(1)
})

setTimeout(() => {
if (!start) {
console.error("No data received from the stream.")
process.exit(1)
}

const msElapsed = Date.now() - start
const sElapsed = msElapsed / 1000
const rate = n / sElapsed

console.log("intake rate:", rate, "transactions/sec")

process.exit(0)
}, 30 * 1000)
```

Results after 30 seconds:
```
intake rate: 223.4826808496314 transactions/sec
```

I acknowledge the challenge in replicating this scenario due to our application's closed-source nature. Unfortunately, our ability to share further details is limited.

### Environment
- Operating System: Windows 11 (16c/32t, 64GB RAM)
- Node.js Version: 20.10.1
- Also appears on Ubuntu 22 server (160 cores, 350GB RAM), with same node version.
- Docker is not being used.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.