hasura / hasura/graphql-engine
migration and subscriptions deadlock
- Dominant language
- TypeScript
- Stars
- 32.1k
- Forks
- 3k
- PR merge metrics
- PR metrics pending
Description
### Is your proposal related to a problem?
We currently use subscriptions to automatically pull in data for a webpage. The subscription runs the query every second. Recently we've discovered it's not possible to add/remove columns from the table the subscription is querying, because the migration results in deadlocks. The error that we're getting is similar to the following (anonymized table names):
Deadlock error:
```
2022-06-16 11:13:22.353 UTC [1324282]: [2-1] db=postgres,user=postgres DETAIL: Process 1324282 waits for AccessExclusiveLock on relation 17965 of database 16426; blocked by process 1324215.
Process 1324215 waits for AccessShareLock on relation 17957 of database 16426; blocked by process 1324282.
```
Migration query:
```
Process 1324282: ALTER TABLE "foo" ADD COLUMN "bar" boolean DEFAULT false NOT NULL
```
Subscription query:
```
Process 1324215: SELECT "_subs"."result_id" , "_fld_resp"."root" AS "result" FROM UNNEST(($1)::uuid[], ($2)::json[]) AS "_subs"("result_id", "result_vars") LEFT OUTER JOIN LATERAL (SELECT json_build_object('foo', "foo"."root" ) AS "root" FROM (SELECT coalesce(json_agg("root" ORDER BY "root.or.foo.or.bar.pg.date" ASC NULLS LAST), '[]' ) AS "root" FROM (SELECT "_14_root.or.foo"."root.or.foo.or.bar.pg.date" AS "root.or.foo.or.bar.pg.date", row_to_json((SELECT "_15_e" FROM (SELECT "_10_root.base"."updated_timestamp" AS "updated_timestamp", 'foo' AS "__typename" ) AS "_15_e" ) ) AS "root" FROM (SELECT * FROM "public"."foo" WHERE ((((("public"."foo"."tenant_id") = ((("_subs"."result_vars"#>>ARRAY['session', 'x-hasura-tenant-id']))::uuid)) OR ((("public"."foo"."tenant_id") IS NULL) AND (((("_subs"."result_vars"#>>ARRAY['session', 'x-hasura-tenant-id']))::uuid) IS NULL)))
```
The problem that's probably causing the deadlocks is that the `ALTER TABLE` statement in the migration query touches every row of the table, which locks until complete. The subscription query contains a `SELECT * FROM "public".foo` statement, which needs to know which columns to return. The subscription query can't return because the columns are being adapted by the migration query.
When analyzing a simple query such as:
```
query Foo($limit: Int, $offset: Int) {
foo(limit: $limit, offset: $offset) {
row_id
}
}
```
I can see the following:
```
SELECT
coalesce(json_agg("root"), '[]') AS "root"
FROM
(
SELECT
row_to_json(
(
SELECT
"_1_e"
FROM
(
SELECT
"_0_root.base"."row_id" AS "row_id"
) AS "_1_e"
)
) AS "root"
FROM
(
SELECT
*
FROM
"public"."foo"
WHERE
(
(
(
("public"."foo"."tenant_id") = (('76ac349d-7b63-4d98-836e-bc557c48a2e9') :: uuid)
)
OR (
(("public"."foo"."tenant_id") IS NULL)
AND (
(('76ac349d-7b63-4d98-836e-bc557c48a2e9') :: uuid) IS NULL
)
)
)
)
) AS "_0_root.base"
) AS "_2_root"
```
If only a single field is requested in the GraphQL query, all fields are queried in SQL as seen in the `SELECT * FROM "public"."foo"` statement. As such, this is problematic when subscriptions are used and we want to run a migration without downtime
### Describe the solution you'd like
If possible, the generated SQL query should avoid `SELECT * FROM table` statements, which would avoid the above-described behavior so that migrations could run without downtime.
### Describe alternatives you've considered
* Run the migration without a default field
* Temporarily disabling the subscription and migrating during that time (with downtime)
### If the feature is approved, would you be willing to submit a PR?
Not sure, depends on the scope of the problem.
Contributor guide
Assessment
This issue has not been assessed yet.