element-hq / element-hq/synapse

Very long query times for get_auth_chain_difference_chains

Open
#17,129 14 comments 17 reactions 0 assignees View on GitHub
A-Database A-Performance O-Occasional S-Major
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

Since upgrading to v1.105.1 because of the security patch, we are seeing very high transaction times specifically for the transaction "get_auth_chain_difference_chains". In our case, we directly upgraded from v1.102.0 to v1.105.1, but heard from others with the same problem after upgrading from v1.104.0.

This could be a regression caused by https://github.com/element-hq/synapse/pull/17044 which modified code called through several indirections by "get_auth_chain_difference_chains" or a regression caused by the security patches.

![image](https://github.com/element-hq/synapse/assets/19955832/ca84c91f-4974-4c34-b3d0-5b4ce4e92ccd)

### Steps to reproduce

- Run a large enough homseserver using Synapse v1.105.1 joined to many rooms
- Observe regular long-running transactions "get_auth_chain_difference_chains"

### Homeserver

kit.edu

### Synapse Version

1.105.1

### Installation Method

Debian packages from packages.matrix.org

### Database

postgresql 13.14-0+deb11u1

### Workers

Multiple workers

### Platform

```
root@main-1:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@main-1:~# uname -a
Linux main-1 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
```

Synapse Main, Workers and the database all run in virtual machines.

### Configuration

Presence is enabled.

### Relevant log output

```shell
I'm not sure which logs would be helpful here and I can't find any relevant log entries in the affected code. Can add logs if you give me a hint which contexts I have to set to debug :)
```

### Anything else that would be useful to know?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.