kestra-io / kestra-io/plugin-debezium
postgres: snapshotMode NEVER rejected by Debezium 3.x — capture/trigger fails to start
- Dominant language
- Java
- Stars
- 5
- Forks
- 11
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 9
Description
## Summary
On the PostgreSQL connector, `Capture`, `Trigger`, and `RealtimeTrigger` configured with `snapshotMode: NEVER` fail to start. Debezium 3.x no longer accepts the `never` snapshot mode for the Postgres connector, so the embedded engine rejects the configuration immediately. `NEVER` is an advertised enum value, so any user selecting it from the UI dropdown hits a hard failure.
## Actual Behaviour
The connector never starts. The embedded engine throws:
```
io.debezium.DebeziumException: Connector configuration is not valid. The 'snapshot.mode' value is invalid: Value must be one of always, initial_only, configuration_based, when_needed, initial, custom, no_data
at io.debezium.embedded.async.AsyncEmbeddedEngine.validateAndGetConnectorConfig(AsyncEmbeddedEngine.java:803)
at io.debezium.embedded.async.AsyncEmbeddedEngine.initializeConnector(AsyncEmbeddedEngine.java:372)
at io.debezium.embedded.async.AsyncEmbeddedEngine.run(AsyncEmbeddedEngine.java:208)
at io.kestra.plugin.debezium.AbstractDebeziumRealtimeTrigger.lambda$publisher$0(AbstractDebeziumRealtimeTrigger.java:173)
```
For a `Capture` this fails the task run. For a `RealtimeTrigger` it produces a continuous retry-loop: the connector repeatedly fails validation, and each torn-down engine thread emits `java.lang.InterruptedException: sleep interrupted` from `PostgresReplicationConnection` keep-alive. No executions are ever created.
## Expected Behaviour
`snapshotMode: NEVER` should start the connector and stream changes without an initial snapshot (the behaviour described in the property docs: "The connector never performs snapshots… continues streaming changes from the stored LSN"). In Debezium 3.x the Postgres connector expresses this as `no_data`, so Kestra should map `NEVER` to the value the connector accepts.
## Root Cause
`PostgresService.java` maps the enum to the Debezium property by lowercasing the enum name:
```java
// plugin-debezium-postgres/.../postgres/PostgresService.java:34
properties.put("snapshot.mode",
runContext.render(postgres.getSnapshotMode())
.as(PostgresInterface.SnapshotMode.class).orElseThrow()
.name().toLowerCase(Locale.ROOT));
```
`PostgresInterface.SnapshotMode` (`plugin-debezium-postgres/.../postgres/PostgresInterface.java:106`) declares `INITIAL, ALWAYS, NEVER, INITIAL_ONLY`. `NEVER` therefore becomes `"never"`, which the Postgres connector rejected starting in Debezium 3.x — the accepted values are `always, initial_only, configuration_based, when_needed, initial, custom, no_data`. The naive `name().toLowerCase()` mapping is stale for the values that don't match 1:1 with the Debezium vocabulary.
Likely fix: translate `NEVER` → `no_data` for the Postgres connector (e.g. an explicit per-enum mapping instead of `name().toLowerCase()`), and audit the other values against the Debezium 3.x Postgres list. Note this is connector-specific — MySQL and others may still accept `never`, so the fix should live in the Postgres mapping, not the shared enum.
## Reproducer
Against a Postgres 16 logical-replication database, either of the following fails to start:
```yaml
id: reproducer_capture
namespace: company.team
tasks:
- id: capture
type: io.kestra.plugin.debezium.postgres.Capture
hostname: localhost
port: "5432"
username: "{{ secret('PG_USERNAME') }}"
password: "{{ secret('PG_PASSWORD') }}"
database: postgres
pluginName: PGOUTPUT
snapshotMode: NEVER
includedTables:
- public.my_table
```
```yaml
id: reproducer_trigger
namespace: company.team
tasks:
- id: log_event
type: io.kestra.plugin.core.log.Log
message: "fired"
triggers:
- id: realtime
type: io.kestra.plugin.debezium.postgres.RealtimeTrigger
database: postgres
hostname: localhost
port: "5432"
username: "{{ secret('PG_USERNAME') }}"
password: "{{ secret('PG_PASSWORD') }}"
pluginName: PGOUTPUT
snapshotMode: NEVER
slotName: kestra
publicationName: kestra_publication
includedTables:
- public.my_table
```
Switching `snapshotMode` to `INITIAL` (or `ALWAYS`) makes both work.
## Logs / Stack Trace
See the `DebeziumException` in **Actual Behaviour** above. For the `RealtimeTrigger`, this repeats on each retry, accompanied by:
```
ERROR connector-kestra_-keep-alive PostgresReplicationConnection Unexpected exception while performing keepalive status update on the replication stream
java.lang.InterruptedException: sleep interrupted
```
## Environment
- **Kestra version**: OSS `develop` (observed during QA)
- **Plugin version**: `plugin-debezium` 1.4.3-SNAPSHOT
- **Debezium**: 3.3.1
- **Database**: PostgreSQL 16
- **Deployment**: OSS self-hosted (local dev)
## Additional Context
Discovered during QA of #174 / PR #176 (the JMX MBean collision fix). The MBean fix itself is unaffected — this is a separate, pre-existing snapshot-mode mapping defect surfaced while exercising the Postgres trigger path.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in plugin-debezium-postgres/.../postgres/PostgresService.java and inspect the SnapshotMode enum in PostgresInterface.java. Compare the enum values with the Debezium 3.x PostgreSQL snapshot.mode values, then use the provided Capture or RealtimeTrigger reproducer to verify that NEVER starts without the validation failure and retry loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql
- Domain
- database
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100