grpc / grpc/grpc-node

Potential memory leak related to grpc health probe

Open
#2,975 2 comments 3 reactions 0 assignees View on GitHub
package: @grpc/grpc-js package: grpc-health-check
Dominant language
TypeScript
Stars
4.8k
Forks
716
Avg merge
2d 3h
Merged PRs (30d)
10

Description

Hey folks I have a very simple grpc health endpoint (https://github.com/grpc-ecosystem/grpc-health-probe) and can observe a memory leak coinciding with the regular health check probes using up roughly half a GB every twelve hours in my specific deployment setting.

In fact I could reproduce it with something along the lines of

```
while true; do sleep 0.001; grpc_health_probe -addr localhost:50051; done
```

where I can reproduce the memory leak very quickly.

The solution I have found was to tune the grpc options and to add

```typescript
'grpc.max_connection_idle_ms': 10000, // Close idle connections after 10s
'grpc.max_connection_age_ms': 30000, // Force connection close after 30s
```

I wanted to raise this here since I have seen multiple folks run into memory leaks and it was hard to find a workaround.

See
- https://github.com/grpc/grpc-node/issues/2068
- https://github.com/grpc-ecosystem/grpc-health-probe/issues/34

Env: Linux, Node v22, @grpc/grpc-js@1.13.0

I believe a memory leak here is unfortunate even if it turns out it's the client disconnecting in wrong ways. Hope this helps.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.