hasura / hasura/graphql-engine
Hasura is not working when using pgpool to connect with RDS master and read replicas endpoint
- Dominant language
- TypeScript
- Stars
- 32.1k
- Forks
- 3k
- PR merge metrics
- PR metrics pending
Description
I used pgpool to forward traffic read/write traffic for RDS master and read replicas but hasura is not working. When i am trying to add db to other pods its working fine also pgpool is up and running, expect for hasura pod its not working at the end of the logs getting HASURA DIED.
Pgpool yaml file -
apiVersion: v1
kind: ConfigMap
metadata:
name: pgpool-config
labels:
app: pgpool-config
data:
pgpool.conf: |-
listen_addresses = '*'
port = 9999
socket_dir = '/var/run/pgpool'
pcp_listen_addresses = '*'
pcp_port = 9898
pcp_socket_dir = '/var/run/pgpool'
listen_backlog_multiplier = 1
backend_hostname0 = 'primaryendpoint.rds.amazonaws.com'
backend_port0 = 5432
backend_weight0 = 0
backend_flag0 = 'ALWAYS_PRIMARY|DISALLOW_TO_FAILOVER'
backend_hostname1 = 'readreplicaendpoint.rds.amazonaws.com'
backend_port1 = 5432
backend_weight1 = 999
backend_flag1 = 'DISALLOW_TO_FAILOVER'
sr_check_period = 0
enable_pool_hba = on
backend_clustering_mode = 'streaming_replication'
num_init_children = 1
max_pool = 2
child_life_time = 300
child_max_connections = 0
connection_life_time = 0
client_idle_limit = 0
connection_cache = off
replication_mode = off
load_balance_mode = on
master_slave_mode = on
ssl = on
log_connections = on
log_hostname = on
log_statement = on
log_per_node_statement = on
failover_on_backend_error = off
pool_passwd: |-
postgres:md5
pool_hba.conf: |-
local all all md5
host all all 127.0.0.1/32 md5
host all all 0.0.0.0/0 md5
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: pgpool
spec:
replicas: 1
selector:
matchLabels:
app: pgpool
template:
metadata:
labels:
app: pgpool
spec:
containers:
- name: pgpool
image: pgpool/pgpool:4.2.2
env:
- name: POSTGRES_USERNAME
value: postgres
- name: POSTGRES_PASSWORD
value: postgres123
volumeMounts:
- name: pgpool-config
mountPath: /config
volumes:
- name: pgpool-config
configMap:
name: pgpool-config
---
apiVersion: v1
kind: Service
metadata:
name: pgpool
spec:
selector:
app: pgpool
ports:
- name: pgpool-port
protocol: TCP
port: 9999
targetPort: 9999
logs -
2021-06-18T17:47:13.709169497Z Connection established, got:
2021-06-18T17:47:13.709184284Z GET query missing.
2021-06-18T17:47:13.709475612Z Internal api is ready
2021-06-18T17:47:13.709688008Z Started hasura with PID 72
2021-06-18T17:47:13.709727180Z Monitoring top 7 processes with `ps auxh`
2021-06-18T17:47:13.711232349Z {"timestamp":"2021-06-18T17:47:13.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_DATABASE_URL"}}
2021-06-18T17:47:13.711932453Z {"timestamp":"2021-06-18T17:47:13.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9001"}}
2021-06-18T17:47:13.712658758Z {"timestamp":"2021-06-18T17:47:13.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 60 for 9001 to be ready"}}
2021-06-18T17:47:14.051798587Z {"type":"startup","timestamp":"2021-06-18T17:47:14.051+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":5},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":4000},"enable_maintenance_mode":false,"enabled_log_types":["http-log","websocket-log","startup","webhook-log","query-log"],"server_host":"HostAny","enable_allowlist":false,"remote_schema_permissions":true,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata"],"enable_telemetry":false,"enable_console":false,"auth_hook":null,"infer_function_permissions":true,"experimental_features":[],"jwt_secret":{"audience":"results-api","claims_namespace_path"
2021-06-18T17:47:14.051856099Z {"type":"startup","timestamp":"2021-06-18T17:47:14.051+0000","level":"info","detail":{"kind":"postgres_connection","info":{"retries":1,"database_url":"postgres://postgres:...@pgpool:9999/database"}}}
2021-06-18T17:47:14.051862246Z {"type":"startup","timestamp":"2021-06-18T17:47:14.051+0000","level":"info","detail":{"kind":"schema-sync","info":{"thread_id":"ThreadId 16","instance_id":"b59aa44e-8c1f-4f39-a564-1b5b19c68215","message":"listener thread started"}}}
2021-06-18T17:47:15.051962667Z {"type":"unstructured","timestamp":"2021-06-18T17:47:14.051+0000","level":"info","detail":"Thread SchemeUpdate.listener (re)started"}
2021-06-18T17:48:07.892932343Z node 7 0.5 0.1 573404 41156 ? Sl 17:47 0:00 node --unhandled-rejections=strict /api.js
2021-06-18T17:48:07.892935132Z node 315 0.0 0.0 2312 692 ? S 17:48 0:00 head -n 7
2021-06-18T17:48:10.905421416Z node 20 1.7 0.1 716792 56508 ? Sl 17:47 0:01 /usr/local/bin/node --unhandled-rejections=strict /api.js
2021-06-18T17:48:10.905445649Z node 32 1.6 0.1 717236 57216 ? Sl 17:47 0:01 /usr/local/bin/node --unhandled-rejections=strict /api.js
2021-06-18T17:48:10.905451218Z node 25 1.6 0.1 716756 56276 ? Sl 17:47 0:00 /usr/local/bin/node --unhandled-rejections=strict /api.js
2021-06-18T17:48:10.905455654Z node 18 1.6 0.1 716772 56440 ? Sl 17:47 0:00 /usr/local/bin/node --unhandled-rejections=strict /api.js
2021-06-18T17:48:10.905473064Z node 76 0.6 0.3 1075315096 108780 ? Sl 17:47 0:00 graphql-engine --database-url postgres://postgres:postgres123@pgpool:9999/database serve --enabled-apis=metadata --server-port=9001
2021-06-18T17:48:10.905499191Z node 7 0.4 0.1 573404 41156 ? Sl 17:47 0:00 node --unhandled-rejections=strict /api.js
2021-06-18T17:48:10.905502706Z node 327 0.0 0.0 2312 692 ? S 17:48 0:00 head -n 7
2021-06-18T17:48:13.858058960Z {"timestamp":"2021-06-18T17:48:13.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"failed waiting for 9001, try increasing HASURA_GRAPHQL_MIGRATIONS_SERVER_TIMEOUT (default: 30)"}}
2021-06-18T17:48:13.908522445Z Hasura died.
Can you anyone help me with this, thank you
Contributor guide
Research direction
Start by reproducing the deployment with the provided pgpool YAML and HASURA_GRAPHQL_DATABASE_URL, then inspect the migrations startup path and the temporary server on port 9001. Compare the pgpool connection behavior with the logged 9001 readiness timeout; done means identifying a reproducible configuration or compatibility cause and documenting the required correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, postgresql
- Domain
- backend, database, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100