element-hq / element-hq/synapse
Bridged apps occasionally resend full message history, causing extreme spam
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
I run several bridges connected to my synapse server (IRC, Signal, and Discord). Sometimes, without any rhyme or reason, all of my historical outgoing messages get replayed on those bridges. It is extremely disruptive because it causes spam on the respective servers and I have to force disconnect the bridges until the spam resolves itself, which can take several hours. This has happened to me 3 times, so it is extremely rare, but nonetheless a giant PITA when it does.
### Steps to reproduce
Unknown. It just seems to happen at random. The logs mention a lost database connection but I have rebooted the database server many times in the past and synapse reconnects just fine and keeps operating normally.
### Homeserver
matrix.xens.org
### Synapse Version
1.131.0 but this has happened on older versions as well
### Installation Method
pip (from PyPI)
### Database
PostgreSQL through pgbouncer. Single server. Have not ported from SQLite. Have not restored from backup.
### Workers
Single process
### Platform
Ubuntu 24.04
Linux matrix 6.8.0-60-generic #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
### Configuration
Nothing unusual other than bridges installed. Since the bug causes *all* bridges to replay messages I cannot see how it would be a problem in a bridge.
### Relevant log output
```shell
2025-06-08 16:15:27,607 - twisted - 279 - CRITICAL - sentinel - Rollback failed
Traceback (most recent call last):
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/threadpool.py", line 285, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
--- ---
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/enterprise/adbapi.py", line 52, in rollback
self._connection.rollback()
psycopg2.InterfaceError: connection already closed
2025-06-08 16:15:27,607 - twisted - 279 - CRITICAL - sentinel - Rollback failed
Traceback (most recent call last):
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/threadpool.py", line 269, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/threadpool.py", line 285, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
--- ---
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/enterprise/adbapi.py", line 287, in _runWithConnection
conn.rollback()
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/enterprise/adbapi.py", line 61, in rollback
self._pool.disconnect(self._connection)
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/enterprise/adbapi.py", line 426, in disconnect
raise Exception("wrong connection for thread")
builtins.Exception: wrong connection for thread
2025-06-08 16:15:27,608 - synapse.metrics.background_process_metrics - 253 - ERROR - prune_old_user_ips-90937 - Background process 'prune_old_user_ips' threw an exception
Traceback (most recent call last):
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/database.py", line 781, in new_transaction
r = func(cursor, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/databases/main/client_ips.py", line 495, in _prune_old_user_ips_txn
txn.execute(sql, (timestamp,))
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/database.py", line 427, in execute
self._do_execute(self.txn.execute, sql, parameters)
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/database.py", line 489, in _do_execute
return func(sql, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/metrics/background_process_metrics.py", line 251, in run
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/databases/main/client_ips.py", line 497, in _prune_old_user_ips
await self.db_pool.runInteraction(
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/database.py", line 953, in runInteraction
return await delay_cancellation(_runInteraction())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/internet/defer.py", line 2010, in _inlineCallbacks
result = context.run(
^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/failure.py", line 549, in throwExceptionIntoGenerator
return g.throw(self.value.with_traceback(self.tb))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/database.py", line 919, in _runInteraction
result: R = await self.runWithConnection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/database.py", line 1048, in runWithConnection
return await make_deferred_yieldable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/threadpool.py", line 269, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/threadpool.py", line 285, in
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/context.py", line 117, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/python/context.py", line 82, in callWithContext
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
result = func(conn, *args, **kw)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/database.py", line 1041, in inner_func
return func(db_conn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/database.py", line 799, in new_transaction
conn.rollback()
File "/home/snex/synapse/env/lib/python3.12/site-packages/synapse/storage/database.py", line 214, in rollback
self.conn.rollback()
File "/home/snex/synapse/env/lib/python3.12/site-packages/twisted/enterprise/adbapi.py", line 66, in rollback
raise ConnectionLost()
twisted.enterprise.adbapi.ConnectionLost
```
### Anything else that would be useful to know?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.