debezium / debezium/dbz

Incremental snapshot failed when table has a generated column

Open
#2,020 3 comments 0 reactions 1 assignee Claimed by @ganeshannt View on GitHub
component/core-library component/postgresql-connector type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Bug report

**What Debezium connector do you use and what version?**

- postgres-connector
- 3.5.1.Final, 3.1.2.Final, 3.6.0.Alpha1

---

**What is the connector configuration?**

```
name=debezium-local
connector.class=io.debezium.connector.postgresql.PostgresConnector
max.batch.size=200000
max.queue.size=400000
poll.interval.ms=500
snapshot.fetch.size=200000
signal.enabled.channels=source
signal.data.collection=public.signals

database.hostname=localhost
database.dbname=postgres
database.port=5432
database.user=postgres
database.password=postgres

publication.autocreate.mode=disabled
publication.name=debezium_publication
plugin.name=pgoutput
slot.name=debezium_slot
topic.prefix=development
snapshot.mode=NO_DATA
```

---

**What is the captured database version and mode of deployment?**

- postgres:17
- local / docker

---

**What behavior do you expect?**

Correctly execute incremental snapshot after signal:
```
INSERT INTO signals (id, type, data) VALUES (gen_random_uuid(), 'execute-snapshot', '{"type":"incremental","data-collections":["public.data_generated"]}');
```

---

**What behavior do you see?**

```
2026-06-03 16:56:55,868 INFO [io.deb.pip.sig.act.sna.ExecuteSnapshot] (debezium-postgresconnector-shard_1-change-event-source-coordinator) Requested 'INCREMENTAL' snapshot of data collections '[public.data_generated]' with additional conditions '[]' and surrogate key 'PK of table will be used'
2026-06-03 16:56:55,883 WARN [io.deb.con.pos.PostgresSchema] (debezium-postgresconnector-shard_1-change-event-source-coordinator) Refresh of public.data_generated was requested but the table no longer exists
2026-06-03 16:56:55,885 WARN [io.deb.pip.sig.SignalProcessor] (debezium-postgresconnector-shard_1-change-event-source-coordinator) Action execute-snapshot failed. The signal SignalRecord{id='78bcdc03-f4a1-4f30-aa06-0674b2a8da7f', type='execute-snapshot', data='{"type":"incremental","data-collections":["public.data_generated"]}', additionalData={}} may not have been processed.: io.debezium.DebeziumException: Database error while executing incremental snapshot for table 'CollectionId{id=public.data_generated, additionalCondition=, surrogateKey=}'
at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:349)
at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.addDataCollectionNamesToSnapshot(AbstractIncrementalSnapshotChangeEventSource.java:491)
at io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot.arrived(ExecuteSnapshot.java:78)
at io.debezium.pipeline.signal.SignalProcessor.processSignal(SignalProcessor.java:191)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at io.debezium.pipeline.signal.SignalProcessor.lambda$processSourceSignal$4(SignalProcessor.java:155)
at io.debezium.pipeline.signal.SignalProcessor.executeWithSemaphore(SignalProcessor.java:165)
at io.debezium.pipeline.signal.SignalProcessor.processSourceSignal(SignalProcessor.java:149)
at io.debezium.pipeline.EventDispatcher$2.changeRecord(EventDispatcher.java:301)
at io.debezium.relational.RelationalChangeRecordEmitter.emitCreateRecord(RelationalChangeRecordEmitter.java:79)
at io.debezium.relational.RelationalChangeRecordEmitter.emitChangeRecords(RelationalChangeRecordEmitter.java:47)
at io.debezium.connector.postgresql.PostgresChangeRecordEmitter.emitChangeRecords(PostgresChangeRecordEmitter.java:94)
at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:281)
at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processReplicationMessages(PostgresStreamingChangeEventSource.java:344)
at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.lambda$processMessages$0(PostgresStreamingChangeEventSource.java:236)
at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.decodeInsert(PgOutputMessageDecoder.java:432)
at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.processNotEmptyMessage(PgOutputMessageDecoder.java:208)
at io.debezium.connector.postgresql.connection.AbstractMessageDecoder.processMessage(AbstractMessageDecoder.java:41)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.deserializeMessages(PostgresReplicationConnection.java:723)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.readPending(PostgresReplicationConnection.java:715)
at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.processMessages(PostgresStreamingChangeEventSource.java:236)
at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:187)
at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:42)
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:322)
at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: Column 'generated_column' not found in result set 'id, generated_column, field1, field2, field3' for table 'public.data_generated', columns: {
id int8(19, 0) NOT NULL DEFAULT VALUE nextval('data_generated_id_seq'::regclass)
field1 text(2147483647, 0) DEFAULT VALUE NULL
field2 text(2147483647, 0) DEFAULT VALUE NULL
field3 text(2147483647, 0) DEFAULT VALUE NULL
}
primary key: [id]
Caused by: java.lang.IllegalArgumentException: Column 'generated_column' not found in result set 'id, generated_column, field1, field2, field3' for table 'public.data_generated', columns: {

default charset: null
comment: null
attributes: {
}
. This might be caused by DBZ-4350
at io.debezium.util.ColumnUtils.toArray(ColumnUtils.java:55)
at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.lambda$readChunk$0(AbstractIncrementalSnapshotChangeEventSource.java:286)
at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:649)
at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:515)
at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:280)
... 42 more
```

---

**Do you see the same behaviour using the latest released Debezium version?**

Yes, checked versions:
- 3.6.0.Alpha1
- 3.5.1.Final
- 3.1.2.Final

---

**Do you have the connector logs, ideally from start till finish?**

_You might be asked later to provide [DEBUG/TRACE](https://debezium.io/documentation/reference/stable/operations/logging.html) level log._

Can provide them if needed

---

**How to reproduce the issue using our [tutorial](https://github.com/debezium/debezium-examples/tree/main/tutorial) deployment?**

Create table with generated column and insert some data:
```
CREATE TABLE public.data_generated
(
id BIGSERIAL PRIMARY KEY,
generated_column INT GENERATED ALWAYS AS (id % 2) STORED,
field1 TEXT,
field2 TEXT,
field3 TEXT
);

INSERT INTO data_generated (field1, field2, field3) SELECT i::text, i::text, i::text from generate_series(1, 100) i;
```
Then trigger incremental snapshot (assuming that signal.channel=source and signal.table.name=signals):
```
INSERT INTO signals (id, type, data) VALUES (gen_random_uuid(), 'execute-snapshot', '{"type":"incremental","data-collections":["public.data_generated"]}');
```

**Additional context**
Blocking snapshot works well

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.