ConduitIO / ConduitIO/conduit-connector-postgres

Feature: PostgreSQL table name differentiation

Open
#165 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Go
Stars
19
Forks
12
Avg merge
12h 20m
Merged PRs (30d)
5

Description

### Bug description

Attempted to replicate data records from a table named `orders` to another table in the same database and schema. I assumed that I could accomplish this by setting the destination configuration `table` value to `orders_moved` (which did not yet exist).

I expected the destination connector to check for `orders_moved` and if it did not exist, to create it, and begin replicating data records into the new table created `orders_moved`. Instead, I saw no results. The destination connector flipped to `degraded`. I looked up the logs on our platform to find the following error:

```
error while nacking message: failed to write message to DLQ: DLQ nack threshold exceeded (0/1), original error: failed to execute query for record 0: ERROR: relation "orders_moved" does not exist (SQLSTATE 42P01)
```

I had to create the table `orders_moved` in the database in order for records to replicate successfully.

This was just a simple example.

**Which begs the question:** How would this work with multiple tables? Could there be a `_prefix` tag in the destination config? I would expect some way to be able to differentiate the table in the event the table might already exist in the destination.

### Steps to reproduce

1. Create an application in the test tenant.
2. Configure a PostgreSQL source connector with a table that contains data records.
3. Configure a PostgreSQL destination connector, set `table` value to a table of choice that does not exist.
4. Deploy the application.
5. Check the Datadog logs for the test tenant.

### Version

v0.7.3

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files, tests, or entry points are named. Reproduce the case with a PostgreSQL source and destination using a nonexistent destination table, then inspect the connector's handling of the table configuration and the reported relation error. The desired differentiation mechanism and behavior for multiple tables still need clarification before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.