ConduitIO / ConduitIO/conduit-connector-postgres
Postgres table in which data exists can't be upsert unless "key" is specified in the config when creating
- Dominant language
- Go
- Stars
- 19
- Forks
- 12
- Avg merge
- 12h 20m
- Merged PRs (30d)
- 5
Description
Postgres table in which data exists can't be upsert unless "key" is specified in the config when creating Postgres Destination connector (request: /v1/connectors). Even if the "key" field is present in the metadata
Steps to reproduce
1. There is a table with data with the primary "key" in Postgres
2. Create a new pipeline
3. Create Postgres destination connector with configs
"settings": {
"url": "postgresql://postgres:postgres@localhost:5432/postgres",
table: "",
"key": ""
}
4. Create Source connector that transfers the "key" in metadata (For example, Postgres or Snowflake Source)
5. Run the pipeline -> the system is trying to transfer data by "key" which already exists
Actual result: data isn't transferred to the Postgres table
Expected result: The data should be updated by "key" from the metadata.
2022-05-12T13:56:39+00:00 ERR node stopped error="node 9873f4b8-61d7-483e-8b24-a52f12395ad0-acker stopped with error: error receiving ack: ERROR: duplicate key value violates unique constraint \"table2_pkey\" (SQLSTATE 23505)" component=pipeline.Service node_id=9873f4b8-61d7-483e-8b24-a52f12395ad0-acker stack=[{"file":"/Users/serhii/conduit/pkg/pipeline/lifecycle.go","func":"[github.com/conduitio/conduit/pkg/pipeline.(*Service](http://github.com/conduitio/conduit/pkg/pipeline.(*Service)).runPipeline.func1","line":387},{"file":"/Users/serhii/conduit/pkg/connector/destination.go","func":"[github.com/conduitio/conduit/pkg/connector.(*destination](http://github.com/conduitio/conduit/pkg/connector.(*destination)).Ack","line":194},{"file":"/Users/serhii/conduit/pkg/plugin/builtin/v1/destination.go","func":"[github.com/conduitio/conduit/pkg/plugin/builtin/v1.(*destinationPluginAdapter](http://github.com/conduitio/conduit/pkg/plugin/builtin/v1.(*destinationPluginAdapter)).Ack","line":137}]
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the existing-row upsert failure using the PostgreSQL destination request with an empty key and metadata containing the key. Start with the error path mentioned in pkg/pipeline/lifecycle.go, pkg/connector/destination.go, and pkg/plugin/builtin/v1/destination.go; done means the existing row is updated by the metadata key without a duplicate-key error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100