hasura / hasura/graphql-engine

Hasura Admin UI can be very slow when run inside minikube / Kubernetes

Open
#10,215 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

### Discussed in https://github.com/hasura/graphql-engine/discussions/10214

Originally posted by **delaurentis** April 23, 2024
**Background**

- In production, we run Hasura in Kubernetes / GKE inside Google Cloud.
- For local development, we mirror this setup, by running Hasura inside Kuberntes / minikube.

**Problem**

We've noticed that using the Hasura Admin UI can become very slow (30 seconds - 1 minute for pages to load). Some operations, like applying permissions can take 10+ minutes. This happens even on high end development machines with lots of RAM / CPU power (16GB, with 8GB reserved to running Kubernetes locally with Minikube)

When we run Hasura outside of Kubernetes, Hasura Admin UI is fast (pages load quickly, and applying permissions happens in <1 minute).

**Advice Needed :)**

Any tips for improving Hasura's performance with Kubernetes or debugging why it is so slow?

Here's our Kubernetes stateful set definition for Hasura for reference (containers are created for both postgres and hasura)

```
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgresdb
spec:
serviceName: 'postgresdb'
replicas: 1
selector:
matchLabels:
app: postgresdb
template:
metadata:
labels:
app: postgresdb
spec:

containers:
- name: postgresdb
image:
imagePullPolicy: IfNotPresent
args:
- -c
- hba_file=/etc/postgres/conf/pg_hba.conf
- -c
- config_file=/etc/postgres/conf/postgresql.conf
- -c
- jit=off
ports:
- containerPort: 5432
volumeMounts:
- name: db-data
mountPath: /home/postgres/pgdata
- name: db-certs
mountPath: /etc/postgres/ssl
- name: db-config
mountPath: /etc/postgres/conf
- name: hasura
image: hasura/graphql-engine:v2.15.2
ports:
- containerPort: 8080
env:
- name: ACTION_BASE_URL
value: http://host.minikube.internal:3000
- name: HASURA_GRAPHQL_METADATA_DATABASE_URL
value: postgres://postgres:postgres@localhost:5432/tsdb
- name: HASURA_GRAPHQL_DATABASE_URL
value: postgres://postgres:postgres@localhost:5432/tsdb
- name: HASURA_GRAPHQL_TS_DATABASE_URL
value: postgres://postgres:postgres@localhost:5432/tsdb
- name: HASURA_GRAPHQL_ENABLE_CONSOLE
value: 'true'
- name: HASURA_GRAPHQL_DEV_MODE
value: 'true'
- name: HASURA_GRAPHQL_ENABLED_LOG_TYPES
value: startup, http-log, webhook-log, websocket-log, query-log
- name: PGSSLMODE
value: disable
- name: HASURA_GRAPHQL_ADMIN_SECRET
value:
- name: HASURA_GRAPHQL_JWT_SECRET
value: '{"type":"HS256","key":""}'
volumes:
- name: db-readonly-certs
secret:
secretName: db-tls-secret
- name: db-config
configMap:
name: postgres-config
- name: db-certs
emptyDir: {}
- name: log
emptyDir: {}

volumeClaimTemplates:
- metadata:
name: db-data
spec:
accessModes: ['ReadWriteOnce']
storageClassName: 'standard'
resources:
requests:
storage: 10Gi
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the slowdown with the provided Kubernetes StatefulSet and compare Admin UI page loads and permission application outside Kubernetes. No repository file or test is named; done would require identifying the Kubernetes-related cause and documenting a verified remediation.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgres
Domain
databases, infrastructure, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.