hasura / hasura/graphql-engine

Hasura Engine RAM consumption grows indefinitely

Open
#10,056 4 comments 2 reactions 0 assignees View on GitHub
k/bug
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

### Version Information
VERSION=v2.33.3
Server Version:
VERSION=v2.33.3

### Environment
Kubernetes deployment.
Nothing special in charts or values...

### What is the current behaviour?
While CPU consumption resembles the actual load, the RAM consumption grows indefinitely
![image](https://github.com/hasura/graphql-engine/assets/108898880/e71d8198-c4aa-4c2d-962b-ea6cdf6017ec)
Such behaviour became apparent after HPA limits were raised. Otherwise, K8s just kills pods very frequently.
Such behaviour drives managed Postgres instance crazy too.
![image](https://github.com/hasura/graphql-engine/assets/108898880/03e827f0-a2b9-4e94-9be8-0f8ea7850cc7)

### What is the expected behaviour?
Service resource consumption rise and fall together with load

### How to reproduce the issue?
Deploy hasura instance in K8s and apply intermittent load. Wait for a couple of days.

### Any possible solutions/workarounds you're aware of?
We have been struggling with issues related to Hasura stability for some time. We tried to adjust queries and parameters in our services but it did not change the picture. Pods were dying.
However, after we significantly raised HPA limits for Hasura, pods stopped dying but our database management started to go in 'failover mode' out of blue and graph started showing such steady consumption growth.
As of now, the workaround is to kill pods as soon as they become 3 days old.

Please consider fixing it.
Also, are there average numbers for Hasura consumption and other metrics.

Thank you!

### Keywords

RAM, resource consumption

Contributor guide

Open the contributing guide

Research direction

Reproduce v2.33.3 in a Kubernetes deployment with intermittent load over several days, while monitoring Hasura RAM, pod restarts, HPA behavior, and managed Postgres failovers. Done means resource consumption rises and falls with load rather than growing indefinitely, with a focused regression check for the observed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.