hasura / hasura/graphql-engine

Hasura HealthCheck is throwing status 500 on AWS Aurora Serverless (Postgres)

Open
#8,346 4 comments 1 reaction 0 assignees View on GitHub
k/bug
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

### Version Information

Server Version: hasura/graphql-engine:v2.2.1.cli-migrations-v3 Docker Image

### Environment

OSS

### What is the expected behaviour?

On automatic scaling of the DB in Aurora Serverless everything keeps working.

### Keywords

"Aurora Serverless"

### What is the current behaviour?

The health endpoints returns a status of 500. Users cannot fully use hasura.

### How to reproduce the issue?

1. Connect Hasura to Serverless Aurora in AWS
2. Force a scale up/scale down in RDS Serverless to bigger/smaller instance

### Please provide any traces or logs that could help here.

```
{"status":500,"http_version":"HTTP/1.1","url":"/healthz","ip":"10.0.154.222","method":"GET","content_encoding":null}}}
{"type":"http-log","timestamp":"2022-03-14T09:53:15.571+0000","level":"error","detail":{"operation":{"error":{"path":"$","error":"ERROR","code":"unexpected"},"request_id":"910e7dcd-f07a-4b10-9ac9-37288c39e1e5","response_size":48,"raw_query":""},"request_id":"910e7dcd-f07a-4b10-9ac9-37288c39e1e5","http_info":{"status":500,"http_version":"HTTP/1.1","url":"/healthz","ip":"10.0.154.222","method":"GET","content_encoding":null}}}
{"type":"scheduled-trigger","timestamp":"2022-03-14T09:53:24.668+0000","level":"error","detail":{"internal":{"statement":"BEGIN ISOLATION LEVEL REPEATABLE READ ","prepared":true,"error":{"exec_status":"FatalError","hint":null,"message":"current transaction is aborted, commands ignored until end of transaction block","status_code":"25P02","description":null},"arguments":[]},"path":"$","error":"postgres tx error","code":"postgres-error"}}
```

### Any possible solutions?

Changing the HASURA_GRAPHQL_SCHEMA_SYNC_POLL_INTERVAL to 0 did make the issue a bit better, the issue is appearing less often after this was set however the issue is still appearing.

Contributor guide

Open the contributing guide

Research direction

Start with the /healthz endpoint and the logged scheduled-trigger transaction errors while reproducing an Aurora Serverless scale-up or scale-down. Check how HASURA_GRAPHQL_SCHEMA_SYNC_POLL_INTERVAL affects the failure; done means health checks remain non-500 and scheduled triggers recover during scaling.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, postgres
Domain
backend, cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.