hasura / hasura/graphql-engine

Hasura down when a connected PG replica is unavailable, despite connected to main PG DB & another replica

Open
#7,829 0 comments 1 reaction 0 assignees View on GitHub
k/bug
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

### Version Information

Server Version: hasuraci/graphql-engine-pro:v1.3.3-pro.3-read-replica-supported

### Environment

GCP cloudSQL (postgres instance + 2 replicas) connected to Hasura v1.3.3 instance in k8s cluster in the same GCP project

### What is the expected behaviour?

When a cloudsql PG DB replica instance undergoes maintenance and becomes unavailable, Hasura is expected to remain operational since it is still connected to (1) main cloudsql PG DB instance and (2) a second replica instance that is operational and not undergoing maintenance.

### Keywords

Hasura, Google Cloud Platform, GCP, cloudSQL, postgres, replica, maintenance, data connection

### What is the current behaviour?

Hasura v1.3.3 instance returned 503 error while any of its connected DB instances are undergoing maintenance and unavailable (last occurrence: 3mins downtime)

### How to reproduce the issue?

1. Connect Hasura v1.3.3 instance to 1 main PG cloudsql instance and 2 replica instances.
2. on one of the replicas: schedule maintenance or restart
3. query Hasura during replica DB maintenance or restart window.

### Any possible solutions?

Have Hasura remain operational given at least one operational DB connection, and send requests to available DB instances.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Reproduce the failure with one primary and two replica PostgreSQL connections while restarting a replica, then trace connection health and request routing; done means Hasura remains available and uses the operational database connections.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, kubernetes, postgres
Domain
backend, cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.