Hasura crashing often on Fargate. Appears related to "Unexpected exception raised in websocket" errors.

Version Information

Server Version: v2.2.1

Environment

AWS Fargate

What is the expected behaviour?

For my production Hasura server to not crash and continue to be available.

Keywords

"Unexpected exception raised in websocket"
There are other much older bug reports related to this error but no solutions were offered in those reports. They also didn't seem to be crashing a production server.

What is the current behaviour?

The server is crashing and returns a 502/503/504 error 21 times per day for a few minutes each crash. This appears to coincide with thousands of websocket errors:

{
"type": "unstructured",
"timestamp": "2022-02-26T19:44:00.559+0000",
"level": "error",
"detail": "Unexpected exception raised in websocket. Please report this as a bug: Thread killed by timeout manager"
}

I have seen these errors when running locally at times as well, usually when attempting to shut down the Hasura service. I would have to wait a few minutes for Hasura to shut down then I would see hundreds of these errors locally. It seems like these errors may be freezing up my Hasura instance.

How to reproduce the issue?

This happens unpredictably. Sometimes when a user logs in, sometimes when a user attempts to visit their cart, sometimes when I attempt to load the console. it doesn't always happen when I do those things, but I have seen those trigger a crash. It happens more often with more traffic. As much as 21 times in 24 hours.

Screenshots or Screencast

Websocket errors in cloudwatch. Usually there are more than 1000 of these errors that happen at the same time.

Memory and CPU utilization do not reach 100%:

Any possible solutions?

I tried increasing CPU and Memory. I upgraded to the latest stable version. I waited a day after each and still saw the same consistent crash pattern.

I considered performance tuning for subscriptions as documented here - https://hasura.io/docs/latest/graphql/core/databases/postgres/subscriptions/execution-and-performance.html - but I'm unsure how adjusting as described would make much difference since the system isn't under high load and isn't maxing out CPU or Memory.

No source files or tests are named. Start by reviewing the CloudWatch websocket error logs alongside the reported Fargate crashes and reproduction conditions; done requires identifying the cause of the production outage and verifying that the websocket failures no longer crash the server.