kestra-io / kestra-io/plugin-debezium

postgres: snapshotMode NEVER rejected by Debezium 3.x — capture/trigger fails to start

Open Beginner friendly
#178 0 comments 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
5
Forks
11
Avg merge
2d 5h
Merged PRs (30d)
9

Description

## Summary

On the PostgreSQL connector, `Capture`, `Trigger`, and `RealtimeTrigger` configured with `snapshotMode: NEVER` fail to start. Debezium 3.x no longer accepts the `never` snapshot mode for the Postgres connector, so the embedded engine rejects the configuration immediately. `NEVER` is an advertised enum value, so any user selecting it from the UI dropdown hits a hard failure.

## Actual Behaviour

The connector never starts. The embedded engine throws:

```
io.debezium.DebeziumException: Connector configuration is not valid. The 'snapshot.mode' value is invalid: Value must be one of always, initial_only, configuration_based, when_needed, initial, custom, no_data
at io.debezium.embedded.async.AsyncEmbeddedEngine.validateAndGetConnectorConfig(AsyncEmbeddedEngine.java:803)
at io.debezium.embedded.async.AsyncEmbeddedEngine.initializeConnector(AsyncEmbeddedEngine.java:372)
at io.debezium.embedded.async.AsyncEmbeddedEngine.run(AsyncEmbeddedEngine.java:208)
at io.kestra.plugin.debezium.AbstractDebeziumRealtimeTrigger.lambda$publisher$0(AbstractDebeziumRealtimeTrigger.java:173)
```

For a `Capture` this fails the task run. For a `RealtimeTrigger` it produces a continuous retry-loop: the connector repeatedly fails validation, and each torn-down engine thread emits `java.lang.InterruptedException: sleep interrupted` from `PostgresReplicationConnection` keep-alive. No executions are ever created.

## Expected Behaviour

`snapshotMode: NEVER` should start the connector and stream changes without an initial snapshot (the behaviour described in the property docs: "The connector never performs snapshots… continues streaming changes from the stored LSN"). In Debezium 3.x the Postgres connector expresses this as `no_data`, so Kestra should map `NEVER` to the value the connector accepts.

## Root Cause

`PostgresService.java` maps the enum to the Debezium property by lowercasing the enum name:

```java
// plugin-debezium-postgres/.../postgres/PostgresService.java:34
properties.put("snapshot.mode",
runContext.render(postgres.getSnapshotMode())
.as(PostgresInterface.SnapshotMode.class).orElseThrow()
.name().toLowerCase(Locale.ROOT));
```

`PostgresInterface.SnapshotMode` (`plugin-debezium-postgres/.../postgres/PostgresInterface.java:106`) declares `INITIAL, ALWAYS, NEVER, INITIAL_ONLY`. `NEVER` therefore becomes `"never"`, which the Postgres connector rejected starting in Debezium 3.x — the accepted values are `always, initial_only, configuration_based, when_needed, initial, custom, no_data`. The naive `name().toLowerCase()` mapping is stale for the values that don't match 1:1 with the Debezium vocabulary.

Likely fix: translate `NEVER` → `no_data` for the Postgres connector (e.g. an explicit per-enum mapping instead of `name().toLowerCase()`), and audit the other values against the Debezium 3.x Postgres list. Note this is connector-specific — MySQL and others may still accept `never`, so the fix should live in the Postgres mapping, not the shared enum.

## Reproducer

Against a Postgres 16 logical-replication database, either of the following fails to start:

```yaml
id: reproducer_capture
namespace: company.team

tasks:
- id: capture
type: io.kestra.plugin.debezium.postgres.Capture
hostname: localhost
port: "5432"
username: "{{ secret('PG_USERNAME') }}"
password: "{{ secret('PG_PASSWORD') }}"
database: postgres
pluginName: PGOUTPUT
snapshotMode: NEVER
includedTables:
- public.my_table
```

```yaml
id: reproducer_trigger
namespace: company.team

tasks:
- id: log_event
type: io.kestra.plugin.core.log.Log
message: "fired"

triggers:
- id: realtime
type: io.kestra.plugin.debezium.postgres.RealtimeTrigger
database: postgres
hostname: localhost
port: "5432"
username: "{{ secret('PG_USERNAME') }}"
password: "{{ secret('PG_PASSWORD') }}"
pluginName: PGOUTPUT
snapshotMode: NEVER
slotName: kestra
publicationName: kestra_publication
includedTables:
- public.my_table
```

Switching `snapshotMode` to `INITIAL` (or `ALWAYS`) makes both work.

## Logs / Stack Trace

See the `DebeziumException` in **Actual Behaviour** above. For the `RealtimeTrigger`, this repeats on each retry, accompanied by:

```
ERROR connector-kestra_-keep-alive PostgresReplicationConnection Unexpected exception while performing keepalive status update on the replication stream
java.lang.InterruptedException: sleep interrupted
```

## Environment

- **Kestra version**: OSS `develop` (observed during QA)
- **Plugin version**: `plugin-debezium` 1.4.3-SNAPSHOT
- **Debezium**: 3.3.1
- **Database**: PostgreSQL 16
- **Deployment**: OSS self-hosted (local dev)

## Additional Context

Discovered during QA of #174 / PR #176 (the JMX MBean collision fix). The MBean fix itself is unaffected — this is a separate, pre-existing snapshot-mode mapping defect surfaced while exercising the Postgres trigger path.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in plugin-debezium-postgres/.../postgres/PostgresService.java and inspect the SnapshotMode enum in PostgresInterface.java. Compare the enum values with the Debezium 3.x PostgreSQL snapshot.mode values, then use the provided Capture or RealtimeTrigger reproducer to verify that NEVER starts without the validation failure and retry loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
database
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.