element-hq / element-hq/synapse

synapse_port_db: IncorrectDatabaseSetup after port was interrupted

Open
#17,596 2 comments 0 reactions 0 assignees View on GitHub
A-Database A-Portdb O-Uncommon S-Major T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

I've been getting an error message after synapse_port_db is interrupted (e.g. due to network issues) during the first run.

I'd expect that it is possible to resume by starting port_db again.

### Steps to reproduce

Run synapse_port_db:
`synapse_port_db --sqlite-database /var/lib/matrix-synapse/homeserver.db.snap --postgres-config homeserver-postgres.yaml`

After something broke (e.g. connection issue), reconnect again.

Postgres is accessed over the network.

### Homeserver

selfhosted

### Synapse Version

matrix-synapse-py3/unknown,unknown,now 1.113.0+bullseye1 amd64

### Installation Method

Debian packages from packages.matrix.org

### Database

Both/Migration state

### Workers

Single process

### Platform

Debian 11

### Configuration

database:
name: psycopg2
args:
(removed user/password/host)
# seconds of inactivity after which TCP should send a keepalive message to the server
keepalives_idle: 10

# the number of seconds after which a TCP keepalive message that is not
# acknowledged by the server should be retransmitted
keepalives_interval: 10

# the number of TCP keepalives that can be lost before the client's connection
# to the server is considered dead
keepalives_count: 3

### Relevant log output

```shell
Preparing psycopg2...
2024-08-21 10:23:54,009 - synapse.storage.prepare_database - 135 - INFO - ('main', 'state'): Checking existing schema version
2024-08-21 10:23:54,024 - synapse.storage.prepare_database - 139 - INFO - ('main', 'state'): Existing schema is 86 (+1 deltas)
2024-08-21 10:23:54,024 - synapse.storage.databases.main - 391 - INFO - Checking database for consistency with configuration...
2024-08-21 10:23:54,027 - synapse.storage.prepare_database - 433 - INFO - Applying schema deltas for v86
2024-08-21 10:23:54,034 - synapse.storage.prepare_database - 568 - INFO - Schema now up to date
2024-08-21 10:23:54,040 - synapse.metrics - 121 - WARNING - synapse_background_update_status already registered, reregistering
2024-08-21 10:23:54,380 - synapse.storage.util.id_generators - 98 - INFO - Initialising stream generator for push_rules(id): 25
2024-08-21 10:23:54,383 - synapse.storage.util.id_generators - 98 - INFO - Initialising stream generator for push_rules_enable(id): 28
2024-08-21 10:23:54,445 - synapse_port_db - 926 - ERROR -
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 726, in run
self.postgres_store = self.build_db_store(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 642, in build_db_store
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs) # type: ignore[arg-type]
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1896, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/client_ips.py", line 84, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/deviceinbox.py", line 975, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/devices.py", line 1613, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 107, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/media_repository.py", line 114, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 2104, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 183, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 1772, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 152, in __init__
self._un_partial_stated_rooms_stream_id_gen = MultiWriterIdGenerator(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/id_generators.py", line 295, in __init__
self._sequence_gen.check_consistency(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup:
Postgres sequence 'un_partial_stated_room_stream_sequence' is inconsistent with associated stream position
of 'un_partial_stated_room_stream' in the 'stream_positions' table.

This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.

A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:

DELETE FROM stream_positions WHERE stream_name = 'un_partial_stated_room_stream';

This will need to be done every time the server is restarted.

Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 726, in run
self.postgres_store = self.build_db_store(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 642, in build_db_store
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs) # type: ignore[arg-type]
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1896, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/client_ips.py", line 84, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/deviceinbox.py", line 975, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/devices.py", line 1613, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 107, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/media_repository.py", line 114, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 2104, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 183, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 1772, in __init__
super().__init__(database, db_conn, hs)
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 152, in __init__
self._un_partial_stated_rooms_stream_id_gen = MultiWriterIdGenerator(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/id_generators.py", line 295, in __init__
self._sequence_gen.check_consistency(
File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/util/sequence.py", line 192, in check_consistency
raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup:
Postgres sequence 'un_partial_stated_room_stream_sequence' is inconsistent with associated stream position
of 'un_partial_stated_room_stream' in the 'stream_positions' table.

This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.

A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:

DELETE FROM stream_positions WHERE stream_name = 'un_partial_stated_room_stream';

This will need to be done every time the server is restarted.

Postgres sequence 'un_partial_stated_room_stream_sequence' is inconsistent with associated stream position
of 'un_partial_stated_room_stream' in the 'stream_positions' table.

This is likely a programming error and should be reported at
https://github.com/matrix-org/synapse.

A temporary workaround to fix this error is to shut down Synapse (including
any and all workers) and run the following SQL:

DELETE FROM stream_positions WHERE stream_name = 'un_partial_stated_room_stream';

This will need to be done every time the server is restarted.
```

### Anything else that would be useful to know?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.