element-hq / element-hq/synapse
/keys/upload throws 500 error when called quickly
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#8805](https://github.com/matrix-org/synapse/issues/8805).
---
This was noticed due to a bug in Element iOS which required purging all one-time-keys. The background is that the client is attempting to call `/keys/claim` for every OTK available on the server. Usually there's something like 50 keys, at some point after 30 or 40 calls this will error and throw a 500 internal server error. This ends up as the sentry issue below:
https://sentry.matrix.org/sentry/synapse-matrixorg/issues/191099/
> SerializationFailure: could not serialize access due to concurrent update
@erikjohnston provided some improvements that could help here:
> Those transactions should get retried, I'm surprised that there is enough contention there to be causing problems
>
> So things that would help here are:
> 1. split up the transaction to make it quicker (and thus less likely to conflict)
> 2. rewrite the functions to be single SQL statements, or otherwise make it safe to use autocommit mode
>
> Which I think is doable in that transaction by moving the loop out and splitting up fetching OTK and fallback key to different transactions
>
> and possibly combining the select/delete into one statement DELETE FROM e2e_one_time_keys_json WHERE .... RETURNING ..., which would allow you to use autocommit mode
CC @manuroe
Contributor guide
Research direction
Start at the /keys/upload and /keys/claim request handlers and trace the transactions that fetch and delete one-time keys. Reproduce rapid key-claim requests and inspect the SerializationFailure; done means repeated calls no longer produce a 500 response from concurrent updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- api, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100