element-hq / element-hq/synapse

/keys/upload throws 500 error when called quickly

Open
#8,805 0 comments 0 reactions 0 assignees View on GitHub
A-Performance z-p2
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#8805](https://github.com/matrix-org/synapse/issues/8805).

---

This was noticed due to a bug in Element iOS which required purging all one-time-keys. The background is that the client is attempting to call `/keys/claim` for every OTK available on the server. Usually there's something like 50 keys, at some point after 30 or 40 calls this will error and throw a 500 internal server error. This ends up as the sentry issue below:

https://sentry.matrix.org/sentry/synapse-matrixorg/issues/191099/

> SerializationFailure: could not serialize access due to concurrent update

@erikjohnston provided some improvements that could help here:

> Those transactions should get retried, I'm surprised that there is enough contention there to be causing problems
>
> So things that would help here are:
> 1. split up the transaction to make it quicker (and thus less likely to conflict)
> 2. rewrite the functions to be single SQL statements, or otherwise make it safe to use autocommit mode
>
> Which I think is doable in that transaction by moving the loop out and splitting up fetching OTK and fallback key to different transactions
>
> and possibly combining the select/delete into one statement DELETE FROM e2e_one_time_keys_json WHERE .... RETURNING ..., which would allow you to use autocommit mode

CC @manuroe

Contributor guide

Open the contributing guide

Research direction

Start at the /keys/upload and /keys/claim request handlers and trace the transactions that fetch and delete one-time keys. Reproduce rapid key-claim requests and inspect the SerializationFailure; done means repeated calls no longer produce a 500 response from concurrent updates.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
api, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.