citusdata / citusdata/pg_shard

Improve error handling during failed modification

Open
#41 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
C
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

`pg_shard`'s modification logic assumes that any total failure is due to something transient that a retry might overcome. In many cases, an `INSERT` or `UPDATE` could fail due to a constraint check, which is _not_ something that a simple retry will overcome without something else changing.

See #31 for an example of what I mean. In its example, the client sees:

``` sql
# WARNING: Bad result from shard1.demo:5432
# DETAIL: Remote message: duplicate key value violates unique constraint "members_id_key_10007"
# WARNING: Bad result from shard8.demo:5432
# DETAIL: Remote message: duplicate key value violates unique constraint "members_id_key_10007"
# ERROR: could not modify any active placements
```

A well-written application might want to handle the uniqueness violation in a special fashion, but all `pg_shard` gives it is a generic error about not being able to modify any placements.

We probably want to try a modification on a placement, then:
- If the error is in the class of things we think a user cares about (constraints, etc.), we fail-fast and throw them the error
- If the error is network related or otherwise "transient", we continue with the remaining shards. If any modification completes, we mark the transient-failure shard as bad

At a higher level, we need to handle modification outcomes in a ternary fashion:
- **Total Success** — the modification completed successfully
- **Application Failure** — the modification returned successfully, but the remote DB raised an error
- **Infrastructure Failure** — the modification didn't even complete, or did so with a network error

Only the third case is deserving of a "could not modify placement" error. In the second we can fail-fast and tell the user what happened.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.