ConduitIO / ConduitIO/conduit-connector-postgres

Error deleting a row with NOT NULL bytea column

Open
#301 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Go
Stars
19
Forks
12
Avg merge
12h 20m
Merged PRs (30d)
5

Description

### Bug description

When deleting a row with a NOT NULL bytea column, the following error can be observed:

```
failed to encode before payload: failed to marshal data with schema: could not marshal into avro: col_bytea_not_null: avro: *avro.null is unsupported for Avro bytes"
```

Debugging this reveals that all the column values in the **record.Payload.Before** map are set to nil.

### Steps to reproduce

1. Create a source and destination table with a NOT NULL bytea column:

```sql
DROP TABLE IF EXISTS bytea_table_source;
CREATE TABLE bytea_table_source (
id bigserial PRIMARY KEY,
col_bytea_not_null bytea NOT NULL
);

DROP TABLE IF EXISTS bytea_table_destination;
CREATE TABLE bytea_table_destination (
id bigserial PRIMARY KEY,
col_bytea_not_null bytea NOT NULL
);
```

2. Run the following pipeline:

```yaml
version: "2.2"
pipelines:
- id: pipeline1
status: running
name: pipeline1
description: Postgres source, file destination
connectors:
- id: postgres-source
type: source
plugin: builtin:postgres
name: source1
settings:
cdcMode: logrepl
tables: bytea_table_source
url: postgresql://meroxauser:meroxapass@localhost/meroxadb?sslmode=disable
- id: pg-destination
type: destination
plugin: builtin:postgres
name: pg-destination
settings:
url: postgresql://meroxauser:meroxapass@localhost/meroxadb?sslmode=disable
table: bytea_table_destination
```

3. Insert a row with: `INSERT INTO bytea_table_source (col_bytea_not_null) VALUES ('aabbcc'::bytea);`
4. Delete with: `delete from bytea_table_source;`

### Version

v0.14.0

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file or test is named. Start by reproducing the deletion with the supplied PostgreSQL schema, pipeline, and SQL statements, then trace the delete event's record.Payload.Before values through the connector. Done means deleting the row no longer produces the Avro bytes encoding error and preserves the deleted row's non-null bytea value.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.