hasura / hasura/graphql-engine

Untracking tables in parallel results in serialization failure at postgres level

Open
#9,956 0 comments 0 reactions 0 assignees View on GitHub
k/bug
Dominant language
TypeScript
Stars
32.1k
Forks
3k
PR merge metrics
PR metrics pending

Description

### Version Information

Server Version: v2.6.1

### Environment

OSS

### What is the current behaviour?

When using the rest api to untrack tables, I get the following error: `serialization failure due to concurrent update`. It seems this happens if I try to untrack tables in parallel.

### What is the expected behaviour?

Untracking in parallel should succeed.

### How to reproduce the issue?

Write a script that that calls the hasura rest api to untrack tables, but do so in parallel without waiting for each individual request to complete.

### Please provide any traces or logs that could help here.

Here's the full log from hasura showing this issue as reported by datadog

```
{
"id": "AgAAAYtoO3sTKL9qHAAAAAAAAAAYAAAAAEFZdG9PNEdjQUFDYU5TQmw0cVNsYWdBQwAAACQAAAAAMDE4YjY4NTMtODdkOC00ZGQzLTg2OWItODNiMzJlNDFhNTlk",
"content": {
"timestamp": "2023-10-25T19:06:50.515Z",
"service": "graphql-engine",
"attributes": {
"level": "error",
"detail": {
"http_info": {
"method": "POST",
"ip": "34.222.178.92",
"http_version": "HTTP/1.1",
"url": "//v1/metadata",
"status": 400
},
"operation": {
"request_mode": "error",
"error": {
"path": "$",
"code": "concurrent-update",
"error": "serialization failure due to concurrent update"
},
"user_vars": {
"x-hasura-role": "admin"
},
"request_id": "cf86a4f6-cf06-487a-bd40-3a91642f3245",
"response_size": 96
},
"request_id": "cf86a4f6-cf06-487a-bd40-3a91642f3245"
},
"type": "http-log",
"timestamp": "2023-10-25T19:06:50.515+0000"
}
}
}
```

### Any possible solutions/workarounds you're aware of?

Make sure to untrack tables in serial.

It seems that this is related to how hasura connects to the underlying postres store since this seems to be a [postgres error](https://pganalyze.com/docs/log-insights/app-errors/U138).

### Keywords

hasura, postgres, track, error

Contributor guide

Open the contributing guide

Research direction

Start by reproducing concurrent table-untrack requests through the v1/metadata REST endpoint, using the parallel script described in the issue and observing the PostgreSQL serialization error. Trace the metadata untracking request path and its database transaction handling; done means parallel untracking succeeds without a concurrent-update failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres
Domain
api, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.