element-hq / element-hq/synapse

Slow user queries for an Application Service leads to persistent events failing to be sent

Open
#17,621 0 comments 0 reactions 0 assignees View on GitHub
A-Application-Service T-Enhancement
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

For some context, https://github.com/element-hq/synapse/issues/17206 was caused by this issue.

When an application service registers to receive persistent events (i.e. messages) from certain users/rooms, Synapse will push those events to the application service. It does this using the [ApplicationServiceHandler](https://github.com/element-hq/synapse/blob/e563e4bdf3e7d55d767e24205e594f520732986d/synapse/handlers/appservice.py#L74).

Different types of persistent events are pushed in different code paths. `m.room.member` events with a `membership` type of `join` have their own path, while all other persistent events are pushed through [`ApplicationServicesHandler._notify_interested_services`](https://github.com/element-hq/synapse/blob/e563e4bdf3e7d55d767e24205e594f520732986d/synapse/handlers/appservice.py#L122).

Within this method, Synapse iterates through each event to be sent and calls `handle_event` with it. During `handle_event`, Synapse blocks on querying [`GET /_matrix/app/v1/users/{userId}`](https://spec.matrix.org/v1.11/application-service-api/#get_matrixappv1usersuserid) on the application service:

https://github.com/element-hq/synapse/blob/e563e4bdf3e7d55d767e24205e594f520732986d/synapse/handlers/appservice.py#L150-L156

Following `self.appservice_api.query_user` down the stack, the HTTP request [has a timeout of 60s](https://github.com/element-hq/synapse/blob/8e1e62c9e010014cf0d46065de21c82a293cf9a1/synapse/http/client.py#L404-L410).

This means that for a slow connection on a user query, Synapse will wait up to 2 minutes per event(!). It's highly possible that more than 0.5 events/min may be generated, thus causing this AS to fall behind.

Furthermore, we seem to be creating one AS txn per event, whereas we should be batching these up:

https://github.com/element-hq/synapse-private/blob/550d760364bf79147046c7c939af59db06ae17f1/synapse/handlers/appservice.py#L202-L206

I suggest we:

* extract and de-duplicate the senders from all the events
* query each user against the AS with a much lower timeout than 60s (5s?)
* send those user queries in parallel to avoid one blocking all the others.

Contributor guide

Open the contributing guide

Research direction

Start in synapse/handlers/appservice.py at ApplicationServicesHandler._notify_interested_services and handle_event, then trace appservice_api.query_user and the 60-second timeout in synapse/http/client.py. Done means sender queries are deduplicated, issued in parallel with a lower timeout, and persistent events are batched without one slow query blocking delivery.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.