element-hq / element-hq/synapse
No backoff on `v1/user/devices` requests for homeservers that have been reset
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#15365](https://github.com/matrix-org/synapse/issues/15365).
---
### Description
I used run a Conduit instance on the domain `matrix.example.com` with a user `@phesch:matrix.example.com` and joined several rooms with that user. However, this was before I knew about `.well-known` redirects, so I eventually decided to nuke that server and set up a new one under `example.com` with a well-known redirect to `matrix.example.com` (so the homeserver's "actual" domain and DNS didn't change, but the name did).
It has been about 18 months since I reset the homeserver and everything works correctly except for the fact that it receives a large amount of request spam from the Synapse homeservers that were in the rooms my old user `@phesch:matrix.example.com` was in.
These servers constantly request `/federation/v1/user/devices/@phesch:matrix.example.com` despite the fact that they always receive a 403 response (since that user obviously doesn't exist). I guess this has something to do with the usual backoff being reset since my homeserver responds normally to other (current) requests.
This behavior has become rather annoying since it amounts to >60000 requests a day and constitutes about 70% of my server's total incoming HTTP traffic.
Note that Dendrites and other Conduits do not show up as performing these requests in my logs.
### Steps to reproduce
- Set up a matrix homeserver (unsure whether it must be a conduit server) on a subdomain
- Make a user and join some rooms
- Delete the homeserver and make a new one with the homeserver itself on the same subdomain, but with a `.well-known` entry so the homeserver's name is the main domain
- Make a user and join some rooms
- Observe that Synapse homeservers keep requesting `v1/user/devices` at a high rate even though it always returns an error
### Homeserver
Many Synapse servers
### Synapse Version
1.70.1 - 1.80.0rc2 are the versions I found in my logs
### Installation Method
Other (please mention below)
### Database
N/A
### Workers
I don't know
### Platform
N/A
### Configuration
N/A
### Relevant log output
```shell
Here's an excerpt of what this looks like in my nginx logs:
- - [
### Anything else that would be useful to know?
_No response_
Contributor guide
Research direction
No source file or test is named. Start by tracing Synapse's federation v1/user/devices request path and its retry/backoff handling, using the repeated 403 log entries as the reproduction signal. Done means requests to a reset homeserver back off instead of continuing at high frequency, with coverage for the persistent 403 case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100