crate / crate/commons-codec

DMS: Observations from replication tests with RDS PostgreSQL

Open
#11 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
2
PR merge metrics
No merged PRs in 30d

Description

While testing out the DMS transformation, I made a few observations I want to share. Each issue still needs to be specified further with proper steps to reproduce, this is more of a brain dump meta collection. All observations were made on a PostgreSQL RDS source.

# DELETE without primary key

Issuing a `DELETE FROM my_table` (aka truncate) on the source fails to replicate with `Unable to invoke DML operation without primary key information`. Currently, a `WHERE` clause with the primary key is required:

https://github.com/daq-tools/commons-codec/blob/e9f17e90bd4ad044cb6e4d942868dd868af26e1b/src/commons_codec/transform/aws_dms.py#L88

# ALTER TABLE ADD COLUMN

I tried an `ALTER TABLE my_table ADD COLUMN last_updated TIMESTAMP DEFAULT`, which was ignored as being an unsupported operation. Subsequent `UPDATE my_table SET last_updated = NOW() WHERE id = 1;` therefore also fail.
What can be a strategy to handle such situations? Do we need to manually migrate the CrateDB table structure?

Similarly, if my PostgreSQL table looks like this:
```sql
CREATE TABLE public.locations (
device_id TEXT NOT NULL PRIMARY KEY,
location TEXT,
attributes JSONB
);
```
I also cannot have the addition of a new JSON property replicated (again no primary key): `UPDATE locations SET attributes['last_updated'] = TO_JSONB(NOW());`

# Data types within OBJECT

If there is a `JSON` in PostgreSQL with a timestamp, such as `{"last_updated": "2024-08-09T11:54:36.815606Z"}`, the `OBJECT` in CrateDB recognizes `last_updated` as `TEXT`. Since there is no type information in the JSON, there is probably nothing the CDC translator can do. For real-world use cases, we need to think of some strategy (run `ALTER TABLE ADD COLUMN` in CrateDB upfront?).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.