hasura / hasura/graphql-engine

cli migrations v2 in ECS, failed waiting for 9691

Open
#9,874 2 comments 0 reactions 0 assignees View on GitHub
k/bug
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

### Version Information

Server Version: hasura/graphql-engine:v2.4.0.cli-migrations-v2
CLI Version (for CLI related issue):

### Environment

AWS ECS

### What is the current behaviour?
My ECS service is starting tasks which are failing. Hasura is trying to start a migrations server on port 9691 and waiting for the server to start, only to time out.

### What is the expected behaviour?

The container should start up and auto apply migrations

### Please provide any traces or logs that could help here.

September 07, 2023 at 14:59 (UTC+12:00) | {"timestamp":"2023-09-07T02:59:47.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"failed waiting for 9691, try increasing HASURA_GRAPHQL_MIGRATIONS_SERVER_TIMEOUT (default: 30)"}} | web
-- | -- | --
September 07, 2023 at 14:57 (UTC+12:00) | {"type":"startup","timestamp":"2023-09-07T02:57:47.719+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"postgres://postgres:...@gasowl-databaseproxy2.proxy-cmakpmwkxocv.ap-southeast-2.rds.amazonaws.com:5432"}}} | web
September 07, 2023 at 14:57 (UTC+12:00) | {"type":"startup","timestamp":"2023-09-07T02:57:47.719+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","enable_maintenance_mode":false,"enabled_log_types":["http-log","websocket-log","startup","webhook-log"],"server_host":"HostAny","websocket_connection_init_timeout":"WSConnectionInitTimeout {unWSConnectionInitTimeout = Seconds {seconds = 3s}}","enable_allowlist":false,"remote_schema_permissions":false,"log_level":"debug","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"v1-boolean-null-collapse":false,"graceful_shutdown_timeout":60,"enabled_apis":["metadata"],"enable_telemetry":false,"enable_console":false,"auth_hook":null,"infer_function_permissions":true,"experimental_features":[],"events_fetch_batch_size":100,"jwt_secret":[{"audience":null,"claims_format":"stringified_json","claims_namespace":"https://hasura.io/jwt/claims","header":null,"jwk_url":"https://cognito-idp.ap-southeast-2.amazonaws.com/ap-southeast-2_6Kzqj22pd/.well-known/jwks.json","issuer":null}],"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"websocket_compression_options":"NoCompression","console_assets_dir":null,"admin_secret_set":true,"port":9691,"websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Seconds {seconds = 5s}}"}}} | web
September 07, 2023 at 14:57 (UTC+12:00) | {"timestamp":"2023-09-07T02:57:47.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 120 for 9691 to be ready"}} | web
September 07, 2023 at 14:57 (UTC+12:00) | {"timestamp":"2023-09-07T02:57:47.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting to 9691"}} | web
September 07, 2023 at 14:57 (UTC+12:00) | {"timestamp":"2023-09-07T02:57:47.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}} | web
September 07, 2023 at 14:57 (UTC+12:00) | {"timestamp":"2023-09-07T02:57:47.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_DATABASE_URL"}}

### Keywords

AWS ECS, auto cli migrations

Contributor guide

Open the contributing guide

Research direction

Start by tracing the migrations-startup flow that starts a temporary server on port 9691, including HASURA_GRAPHQL_MIGRATIONS_SERVER_TIMEOUT and the migrations server port setting. Reproduce the ECS startup with hasura/graphql-engine:v2.4.0.cli-migrations-v2 using the supplied logs; done means the container starts and automatically applies migrations without timing out.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker
Domain
backend, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.