hasura / hasura/graphql-engine

Hasura close the subscription on server restart with 1000 Normal Closure

Open
#9,453 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

c/subscriptions ✋ k/bug t/gql-services
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

Version Information

Server Version: 2.8.0

Environment

OSS

What is the current behaviour?

When the kubernetes pod restart I see the the subscription ends the connection with a 1000 Normal Closure which could be improved. Our frontend lib graphql-ws handles 1000 as a permanent close and won't retry.

What is the expected behaviour?

The lib creator recommended 1001: Going Away, 1012: Service Restart, or 1013: Try Again Later in this scenario.
See it in the following thread: https://github.com/enisdenjo/graphql-ws/discussions/449

How to reproduce the issue?

I run kubectl delete pod resource-name then it come back again because of the replica set.

Screenshots or Screencast
Please provide any traces or logs that could help here.
{"type":"startup","timestamp":"2023-02-21T16:56:59.411+0000","level":"info","detail":{"kind":"event_triggers","info":"unlocking events of source: account"}}
{"type":"unstructured","timestamp":"2023-02-21T16:56:59.411+0000","level":"info","detail":"Stopping immortal processEventQueue thread"}
{"type":"unstructured","timestamp":"2023-02-21T16:56:59.411+0000","level":"info","detail":"Stopping immortal SchemeUpdate.processor thread"}
{"type":"unstructured","timestamp":"2023-02-21T16:56:59.411+0000","level":"info","detail":"Stopping immortal SchemeUpdate.listener thread"}
{"type":"startup","timestamp":"2023-02-21T16:57:06.929+0000","level":"info","detail":{"kind":"server","info":"gracefully shutting down server"}}
{"type":"pg-client","timestamp":"2023-02-21T16:57:06.929+0000","level":"warn","detail":{"message":"postgres connection failed, retrying(0)."}}
{"type":"event-trigger","timestamp":"2023-02-21T16:57:08.029+0000","level":"error","detail":{"internal":"no connection to the server\n","path":"$","error":"connection error","code":"postgres-error"}}
Any possible solutions/workarounds you're aware of?
Keywords

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository file or test is named. Start by tracing the GraphQL subscription WebSocket shutdown path during a Kubernetes pod restart and compare the emitted close code with graphql-ws handling. Done means the restart uses an appropriate non-permanent closure and the behavior is covered by a relevant test.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, kubernetes
Domain
api, backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.