hasura / hasura/graphql-engine
Hasura HealthCheck is throwing status 500 on AWS Aurora Serverless (Postgres)
- Dominant language
- TypeScript
- Stars
- 32.1k
- Forks
- 3k
- PR merge metrics
- PR metrics pending
Description
### Version Information
Server Version: hasura/graphql-engine:v2.2.1.cli-migrations-v3 Docker Image
### Environment
OSS
### What is the expected behaviour?
On automatic scaling of the DB in Aurora Serverless everything keeps working.
### Keywords
"Aurora Serverless"
### What is the current behaviour?
The health endpoints returns a status of 500. Users cannot fully use hasura.
### How to reproduce the issue?
1. Connect Hasura to Serverless Aurora in AWS
2. Force a scale up/scale down in RDS Serverless to bigger/smaller instance
### Please provide any traces or logs that could help here.
```
{"status":500,"http_version":"HTTP/1.1","url":"/healthz","ip":"10.0.154.222","method":"GET","content_encoding":null}}}
{"type":"http-log","timestamp":"2022-03-14T09:53:15.571+0000","level":"error","detail":{"operation":{"error":{"path":"$","error":"ERROR","code":"unexpected"},"request_id":"910e7dcd-f07a-4b10-9ac9-37288c39e1e5","response_size":48,"raw_query":""},"request_id":"910e7dcd-f07a-4b10-9ac9-37288c39e1e5","http_info":{"status":500,"http_version":"HTTP/1.1","url":"/healthz","ip":"10.0.154.222","method":"GET","content_encoding":null}}}
{"type":"scheduled-trigger","timestamp":"2022-03-14T09:53:24.668+0000","level":"error","detail":{"internal":{"statement":"BEGIN ISOLATION LEVEL REPEATABLE READ ","prepared":true,"error":{"exec_status":"FatalError","hint":null,"message":"current transaction is aborted, commands ignored until end of transaction block","status_code":"25P02","description":null},"arguments":[]},"path":"$","error":"postgres tx error","code":"postgres-error"}}
```
### Any possible solutions?
Changing the HASURA_GRAPHQL_SCHEMA_SYNC_POLL_INTERVAL to 0 did make the issue a bit better, the issue is appearing less often after this was set however the issue is still appearing.
Contributor guide
Research direction
Start with the /healthz endpoint and the logged scheduled-trigger transaction errors while reproducing an Aurora Serverless scale-up or scale-down. Check how HASURA_GRAPHQL_SCHEMA_SYNC_POLL_INTERVAL affects the failure; done means health checks remain non-500 and scheduled triggers recover during scaling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, postgres
- Domain
- backend, cloud, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100