hasura / hasura/graphql-engine

v2.44.0.cli-migrations-v3: No retry for postgres database connection, container frozen

Open
#10,595 0 comments 1 reaction 0 assignees View on GitHub
k/bug
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

### Version Information

Server Version:
CLI Version (for CLI related issue): v2.44.0.cli-migrations-v3

### Environment

self-hosted

### What is the current behaviour?

We are running hasura in our kubernetes cluster. We have a postgres DB (it's ephemeral) in a container deployed next to it. On startup, hasura is faster than the postgres DB. So I naturally see the log entry

```
{"error":"connection error","path":"$","code":"postgres-error","internal":"connection to server at \"localhost\" (127.0.0.1), port 5433 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?\nconnection to server at \"localhost\" (::1), port 5433 failed: Cannot assign requested address\n\tIs the server running on that host and accepting TCP/IP connections?\n"}
```

The problem that I have is, that hasura does not _retry_ the postgres connection. There is no additional logging until the hasura gets killed by `HASURA_GRAPHQL_MIGRATIONS_SERVER_TIMEOUT`. Once a new container is spun up by kubernetes, the postgres db is ready and everything works fine.

I don't remember that we had a similar issue before. So it might be a regression issue, because we've been using the same setup for several months now.

The central question here is: _Is there a retry mechanism for the database connection of the temporary server that's being created by the cli-migrations-v3 image?_. From what I can see, there is not. Even when running the (tests)[https://github.com/hasura/graphql-engine/tree/master/packaging/cli-migrations/v3/test] in the hasura repo, I can see the same behavior if the postgres DB is not already available when hasura starts up.

**I am willing to contribute to the project to fix this, if necessary!**

### What is the expected behaviour?

The container retries the DB connection in a (optional: configurable) interval.

### How to reproduce the issue?

1. Use the `docker-compose` file from the (test folder)[https://github.com/hasura/graphql-engine/blob/master/packaging/cli-migrations/v3/test/docker-compose.yaml].
2. Run `docker-compose up`
3. Check the logs of the `hasura` container and see that there is no retry for the database connection
4. See that the container gets killed once `HASURA_GRAPHQL_MIGRATIONS_SERVER_TIMEOUT` has been reached.

### Screenshots or Screencast

### Please provide any traces or logs that could help here.

In the (test)[https://github.com/hasura/graphql-engine/blob/master/packaging/cli-migrations/v3/test/test.sh] section of the image I can see that the postgres DB is spun up _before_ the hasura instance. Maybe it's a coincident, but this might approve my suspicion that there is no repeated check of the DB connection in the hasura instance.

### Any possible solutions/workarounds you're aware of?

Implement polling/retrying for the database connection.

### Keywords

auto-migrate, cli-migrations-v3, database, postgres, metadata

Contributor guide

Open the contributing guide

Research direction

Start with packaging/cli-migrations/v3/test/docker-compose.yaml and test.sh, then run the documented reproduction with PostgreSQL unavailable when the Hasura container starts. Trace the temporary server's connection handling from the resulting logs. Done means the container retries the database connection and remains available until PostgreSQL is ready, rather than stopping at HASURA_GRAPHQL_MIGRATIONS_SERVER_TIMEOUT.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, postgres
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.