element-hq / element-hq/synapse
Slow user queries for an Application Service leads to persistent events failing to be sent
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
For some context, https://github.com/element-hq/synapse/issues/17206 was caused by this issue.
When an application service registers to receive persistent events (i.e. messages) from certain users/rooms, Synapse will push those events to the application service. It does this using the [ApplicationServiceHandler](https://github.com/element-hq/synapse/blob/e563e4bdf3e7d55d767e24205e594f520732986d/synapse/handlers/appservice.py#L74).
Different types of persistent events are pushed in different code paths. `m.room.member` events with a `membership` type of `join` have their own path, while all other persistent events are pushed through [`ApplicationServicesHandler._notify_interested_services`](https://github.com/element-hq/synapse/blob/e563e4bdf3e7d55d767e24205e594f520732986d/synapse/handlers/appservice.py#L122).
Within this method, Synapse iterates through each event to be sent and calls `handle_event` with it. During `handle_event`, Synapse blocks on querying [`GET /_matrix/app/v1/users/{userId}`](https://spec.matrix.org/v1.11/application-service-api/#get_matrixappv1usersuserid) on the application service:
https://github.com/element-hq/synapse/blob/e563e4bdf3e7d55d767e24205e594f520732986d/synapse/handlers/appservice.py#L150-L156
Following `self.appservice_api.query_user` down the stack, the HTTP request [has a timeout of 60s](https://github.com/element-hq/synapse/blob/8e1e62c9e010014cf0d46065de21c82a293cf9a1/synapse/http/client.py#L404-L410).
This means that for a slow connection on a user query, Synapse will wait up to 2 minutes per event(!). It's highly possible that more than 0.5 events/min may be generated, thus causing this AS to fall behind.
Furthermore, we seem to be creating one AS txn per event, whereas we should be batching these up:
https://github.com/element-hq/synapse-private/blob/550d760364bf79147046c7c939af59db06ae17f1/synapse/handlers/appservice.py#L202-L206
I suggest we:
* extract and de-duplicate the senders from all the events
* query each user against the AS with a much lower timeout than 60s (5s?)
* send those user queries in parallel to avoid one blocking all the others.
Contributor guide
Research direction
Start in synapse/handlers/appservice.py at ApplicationServicesHandler._notify_interested_services and handle_event, then trace appservice_api.query_user and the 60-second timeout in synapse/http/client.py. Done means sender queries are deduplicated, issued in parallel with a lower timeout, and persistent events are batched without one slow query blocking delivery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100