airbytehq / airbytehq/airbyte

[source-postgres] NullPointerException on startup when the source database contains an empty enum type

Aperta
#85,295 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
autoteam cdc community connectors/source/postgres hyd-prove hyd-review team/use type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Connector Name

source-postgres

### Connector Version

3.7.2 (3.8.5, the latest release, is affected too — see the version matrix below)

### What step the error happened?

During the sync

### Relevant information

**Summary**

If the source database contains a PostgreSQL enum type declared with **no labels** (`CREATE TYPE my_enum AS ENUM ();`, which is valid PostgreSQL), the embedded Debezium engine throws a `NullPointerException` while priming its type registry and never starts. Every CDC sync on that database then fails.

**Root cause**

This is Debezium [dbz#2525](https://github.com/debezium/dbz/issues/2525). On startup, `PostgresConnectorTask.start()` builds a `PostgresConnection`, whose constructor calls `TypeRegistry.prime()`. That method runs a single query over the whole catalog:

```sql
SELECT t.oid AS oid, t.typname AS name, t.typelem AS element, t.typbasetype AS parentoid,
t.typtypmod as modifiers, t.typcategory as category, e.values as enum_values
FROM pg_catalog.pg_type t
JOIN pg_catalog.pg_namespace n ON (t.typnamespace = n.oid)
LEFT JOIN (SELECT t.enumtypid as id, array_agg(t.enumlabel) as values
FROM pg_catalog.pg_enum t GROUP BY id) e ON (t.oid = e.id)
WHERE n.nspname != 'pg_toast'
```

and then dereferences the array without a null check:

```java
if (CATEGORY_ENUM.equals(category)) {
String[] enumValues = (String[]) rs.getArray("enum_values").getArray(); // <-- NPE
```

An enum type with no labels has `typcategory = 'E'` but no row in `pg_enum`, so the `LEFT JOIN` yields NULL and `.getArray()` throws.

**Why this matters specifically for source-postgres**

Three things make the impact worse than a plain startup crash:

1. **No workaround is available through connector configuration.** `TypeRegistry.prime()` runs inside the `PostgresConnection` constructor, before any table or column filtering is applied, and its query only filters on `nspname != 'pg_toast'`. The offending type does not need to be used by a captured table — or by any table at all. Deselecting columns, deselecting streams, or narrowing the publication changes nothing.

2. **The failure is slow and opaque.** Debezium fails in its own thread, so the connector does not fail fast: it waits out `initial_waiting_seconds` (1200 s at the maximum) before surfacing `Closing the Debezium engine after no records received within 1200 seconds timeout`, and the user-facing message is only *"Something went wrong in the connector."* Each attempt burns 20 minutes, so a single failing job runs for well over an hour across its retries before giving up. Nothing in that output points at an enum type.

3. **The replication slot never advances, so WAL accumulates without bound.** Because no record is ever emitted, the committed LSN stays frozen across every attempt while the database keeps writing. On a busy database this can reach tens of GB of retained WAL within a few hours — and on managed PostgreSQL, where disks grow but never shrink, that turns a connector startup bug into a production storage incident. This is the part that makes it worth fixing rather than documenting.

**Reproduction**

```sql
CREATE TYPE public.empty_enum AS ENUM ();
```

Then run any CDC sync against that database. The type does not have to be referenced by a captured table, or by any table.

Detection query — this is exactly Debezium's own predicate, so every row it returns will crash the connector:

```sql
SELECT t.oid, n.nspname, t.typname, t.typtype, t.typcategory
FROM pg_catalog.pg_type t
JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace
LEFT JOIN (SELECT enumtypid AS id, array_agg(enumlabel) AS vals
FROM pg_catalog.pg_enum GROUP BY 1) ev ON t.oid = ev.id
WHERE n.nspname <> 'pg_toast' AND t.typcategory = 'E' AND ev.vals IS NULL;
```

**Upstream status**

Fixed upstream by [debezium/debezium#7929](https://github.com/debezium/debezium/pull/7929), merged 2026-09-01 — but on `main` only. The maintenance branches (`3.6`, `3.5`, `3.1`) do not carry the commit, so the fix will first ship in Debezium 3.7.0.Final (currently `v3.7.0.Beta1`; the latest Final is `v3.6.2.Final`).

**Version matrix**

| source-postgres | Debezium | Affected |
| --- | --- | --- |
| 3.7.2 | 3.0.1.Final | yes |
| 3.8.5 (latest) | 3.1.2.Final | yes |
| — | 3.7.0.x (unreleased) | fixed |

The Debezium version comes from the BOM in `airbyte-cdk/bulk/toolkits/extract-cdc/build.gradle` (`api platform('io.debezium:debezium-bom:3.1.2.Final')`); `source-postgres/build.gradle` declares `io.debezium:debezium-connector-postgres` without a version, so it resolves from there. Upgrading to the latest source-postgres therefore does not help today.

**Ask**

Bump the Debezium BOM once 3.7.0.Final is released.

In the meantime, a startup precheck would help a lot on its own: running the detection query above during `check`/`discover` and failing immediately with an actionable message would turn an hours-long silent failure with unbounded WAL growth into an instant, self-explanatory error — and it is independent of the Debezium upgrade.

### Relevant log output

```shell
Unable to initialize and start connector's task class 'io.debezium.connector.postgresql.PostgresConnectorTask'
...
CDC events queue poll(): first call - waited 20.00 minutes for events
CDC events queue poll(): returned nothing.
Closing the Debezium engine after no records received within 1200 seconds timeout. Status: receivedFirstRecord=false, hasSnapshotFinished=true, noRecordsAttempts=0
analytics [airbyte/source-postgres:3.7.2] | Type: db-sources-debezium-close-reason | Value: TIMEOUT

ERROR main i.a.c.i.u.ConnectorExceptionHandler(handleException):68 caught exception!
io.airbyte.cdk.integrations.source.relationaldb.state.FailedRecordIteratorException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke "java.sql.Array.getArray()" because the return value of "java.sql.ResultSet.getArray(String)" is null
at io.airbyte.cdk.integrations.source.relationaldb.state.SourceStateIterator.computeNext(SourceStateIterator.kt:38) ~[airbyte-cdk-db-sources-0.48.14.jar:?]
at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.kt:291) ~[airbyte-cdk-core-0.48.14.jar:?]
at io.airbyte.integrations.source.postgres.PostgresSource.main(PostgresSource.java:751) [io.airbyte.airbyte-integrations.connectors-source-postgres.jar:?]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke "java.sql.Array.getArray()" because the return value of "java.sql.ResultSet.getArray(String)" is null
at io.airbyte.cdk.integrations.debezium.internals.DebeziumRecordPublisher.close(DebeziumRecordPublisher.kt:124) ~[airbyte-cdk-db-sources-0.48.14.jar:?]
at io.airbyte.cdk.integrations.debezium.AirbyteDebeziumHandler.getIncrementalIterators$lambda$1(AirbyteDebeziumHandler.kt:108) ~[airbyte-cdk-db-sources-0.48.14.jar:?]
at io.airbyte.cdk.integrations.debezium.internals.DebeziumShutdownProcedure.initiateShutdownProcedure(DebeziumShutdownProcedure.kt:91) ~[airbyte-cdk-db-sources-0.48.14.jar:?]
Caused by: java.lang.NullPointerException: Cannot invoke "java.sql.Array.getArray()" because the return value of "java.sql.ResultSet.getArray(String)" is null
at io.debezium.connector.postgresql.TypeRegistry.createTypeBuilderFromResultSet(TypeRegistry.java:416) ~[debezium-connector-postgres-3.0.1.Final.jar:3.0.1.Final]
at io.debezium.connector.postgresql.TypeRegistry.prime(TypeRegistry.java:379) ~[debezium-connector-postgres-3.0.1.Final.jar:3.0.1.Final]
at io.debezium.connector.postgresql.TypeRegistry.(TypeRegistry.java:132) ~[debezium-connector-postgres-3.0.1.Final.jar:3.0.1.Final]
at io.debezium.connector.postgresql.connection.PostgresConnection.(PostgresConnection.java:118) ~[debezium-connector-postgres-3.0.1.Final.jar:3.0.1.Final]
at io.debezium.connector.postgresql.PostgresConnectorTask.lambda$start$1(PostgresConnectorTask.java:100) ~[debezium-connector-postgres-3.0.1.Final.jar:3.0.1.Final]
at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:99) ~[debezium-connector-postgres-3.0.1.Final.jar:3.0.1.Final]
at io.debezium.embedded.EmbeddedEngine.startSourceTask(EmbeddedEngine.java:640) ~[debezium-embedded-3.0.1.Final.jar:3.0.1.Final]
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:452) ~[debezium-embedded-3.0.1.Final.jar:3.0.1.Final]
```

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.