ConduitIO / ConduitIO/conduit-connector-postgres

Postgres table in which data exists can't be upsert unless "key" is specified in the config when creating

Open
#59 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
19
Forks
12
Avg merge
12h 20m
Merged PRs (30d)
5

Description

Postgres table in which data exists can't be upsert unless "key" is specified in the config when creating  Postgres Destination connector (request: /v1/connectors). Even if the "key" field is present in the metadata
Steps to reproduce

1. There is a table with data with the  primary "key" in Postgres
2. Create a new pipeline
3. Create Postgres destination connector with configs
"settings": {
"url": "postgresql://postgres:postgres@localhost:5432/postgres",
table: "",
"key": ""
}
4. Create Source connector that transfers the "key" in metadata (For example, Postgres or Snowflake Source)
5. Run the pipeline -> the system is trying to transfer data by "key" which already exists
Actual result: data isn't transferred to the Postgres table
Expected result: The data should be updated by "key" from the metadata.

2022-05-12T13:56:39+00:00 ERR node stopped error="node 9873f4b8-61d7-483e-8b24-a52f12395ad0-acker stopped with error: error receiving ack: ERROR: duplicate key value violates unique constraint \"table2_pkey\" (SQLSTATE 23505)" component=pipeline.Service node_id=9873f4b8-61d7-483e-8b24-a52f12395ad0-acker stack=[{"file":"/Users/serhii/conduit/pkg/pipeline/lifecycle.go","func":"[github.com/conduitio/conduit/pkg/pipeline.(*Service](http://github.com/conduitio/conduit/pkg/pipeline.(*Service)).runPipeline.func1","line":387},{"file":"/Users/serhii/conduit/pkg/connector/destination.go","func":"[github.com/conduitio/conduit/pkg/connector.(*destination](http://github.com/conduitio/conduit/pkg/connector.(*destination)).Ack","line":194},{"file":"/Users/serhii/conduit/pkg/plugin/builtin/v1/destination.go","func":"[github.com/conduitio/conduit/pkg/plugin/builtin/v1.(*destinationPluginAdapter](http://github.com/conduitio/conduit/pkg/plugin/builtin/v1.(*destinationPluginAdapter)).Ack","line":137}]

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the existing-row upsert failure using the PostgreSQL destination request with an empty key and metadata containing the key. Start with the error path mentioned in pkg/pipeline/lifecycle.go, pkg/connector/destination.go, and pkg/plugin/builtin/v1/destination.go; done means the existing row is updated by the metadata key without a duplicate-key error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.