element-hq / element-hq/synapse

PostgreSQL database schema incompatible with AWS DMS

Open
#15,583 0 comments 0 reactions 0 assignees View on GitHub
A-Database O-Uncommon S-Tolerable T-Task
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#15583](https://github.com/matrix-org/synapse/issues/15583).

---

**Description:**

AWS DMS (Data Migration Service) is a tool to migrate databases around. It looks like Synapse is incompatible with using it due to some tables with LOB columns which don't have a primary key as per [DMS docs](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Troubleshooting.html#CHAP_Troubleshooting.Oracle.LOBChanges).

It may be a good idea to add the primary keys to the relevant tables to become compatible with AWS DMS, unless there is a technical reason not to do so. The problematic tables are:

![image](https://github.com/matrix-org/synapse/assets/1174866/d588253e-7942-4aec-ad9b-c8a221384d42)

Example DMS log lines for `event_auth`:

>2023-05-11T13:46:37 [SOURCE_UNLOAD ]W: Column 'event_id' was removed from table definition 'public.event_auth': the column data type is LOB and the table has no primary key or unique index (metadatamanager.c:2492)
2023-05-11T13:46:37 [SOURCE_UNLOAD ]W: Column 'auth_id' was removed from table definition 'public.event_auth': the column data type is LOB and the table has no primary key or unique index (metadatamanager.c:2492)
2023-05-11T13:46:37 [SOURCE_UNLOAD ]W: Column 'room_id' was removed from table definition 'public.event_auth': the column data type is LOB and the table has no primary key or unique index (metadatamanager.c:2492)

Contributor guide

Open the contributing guide

Research direction

Begin with the PostgreSQL schema for event_auth and the other tables shown in the linked image, then compare their keys with the AWS DMS LOB requirements and the cited DMS documentation. Confirm whether adding primary keys is technically safe; done means the affected tables are accepted by DMS without the reported SOURCE_UNLOAD warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, postgresql
Domain
cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.