ConduitIO / ConduitIO/conduit-connector-postgres
Error deleting a row with NOT NULL bytea column
- Dominant language
- Go
- Stars
- 19
- Forks
- 12
- Avg merge
- 12h 20m
- Merged PRs (30d)
- 5
Description
### Bug description
When deleting a row with a NOT NULL bytea column, the following error can be observed:
```
failed to encode before payload: failed to marshal data with schema: could not marshal into avro: col_bytea_not_null: avro: *avro.null is unsupported for Avro bytes"
```
Debugging this reveals that all the column values in the **record.Payload.Before** map are set to nil.
### Steps to reproduce
1. Create a source and destination table with a NOT NULL bytea column:
```sql
DROP TABLE IF EXISTS bytea_table_source;
CREATE TABLE bytea_table_source (
id bigserial PRIMARY KEY,
col_bytea_not_null bytea NOT NULL
);
DROP TABLE IF EXISTS bytea_table_destination;
CREATE TABLE bytea_table_destination (
id bigserial PRIMARY KEY,
col_bytea_not_null bytea NOT NULL
);
```
2. Run the following pipeline:
```yaml
version: "2.2"
pipelines:
- id: pipeline1
status: running
name: pipeline1
description: Postgres source, file destination
connectors:
- id: postgres-source
type: source
plugin: builtin:postgres
name: source1
settings:
cdcMode: logrepl
tables: bytea_table_source
url: postgresql://meroxauser:meroxapass@localhost/meroxadb?sslmode=disable
- id: pg-destination
type: destination
plugin: builtin:postgres
name: pg-destination
settings:
url: postgresql://meroxauser:meroxapass@localhost/meroxadb?sslmode=disable
table: bytea_table_destination
```
3. Insert a row with: `INSERT INTO bytea_table_source (col_bytea_not_null) VALUES ('aabbcc'::bytea);`
4. Delete with: `delete from bytea_table_source;`
### Version
v0.14.0
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file or test is named. Start by reproducing the deletion with the supplied PostgreSQL schema, pipeline, and SQL statements, then trace the delete event's record.Payload.Before values through the connector. Done means deleting the row no longer produces the Avro bytes encoding error and preserves the deleted row's non-null bytea value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100