airbytehq / airbytehq/airbyte

[source-postgres] Upgrading to 3.8.X makes Xmin-based streams report "Missing cursor" on schema change review

オープン
#81,431 コメント 4 件 リアクション 0 件 担当者 0 名 GitHub で見る
area/connectors autoteam community connectors/source/postgres needs-triage team/extensibility type/bug
主要言語
Python
スター
22.1k
フォーク
5.3k
PR マージ指標
PR 指標を取得中

説明

### Connector Name

source-postgres

### Connector Version

3.8.0

### What step the error happened?

Other

### Relevant information

## Summary

After upgrading the **source-postgres** connector from **3.7.2** to **3.8.0** (also reproduced on **3.8.1**), a subset of streams configured to use the **Xmin System Column** cursor start reporting **"Missing cursor"** (in red) instead of the expected **``** when reviewing a schema change on the connection.

Because these streams are now flagged as missing a cursor, Airbyte blocks saving the connection until a cursor column is manually selected — even though the streams were previously syncing correctly with the source-defined Xmin cursor and the incoming schema change itself is non-breaking.

This looks like a regression: the same connection and streams behave correctly on 3.7.2, and the issue appears deterministically after upgrading.

## Environment

| Item | Value |
| ------------------------- | -------------------------------------------------------------------------- |
| Airbyte | Self-hosted OSS **1.6.7** |
| Source | Postgres |
| source-postgres connector | Regression appears on **3.8.0** and **3.8.1**; **3.7.2** works as expected |
| Destination | Redshift |
| Source cursor method | **Detect changes with Xmin System Column** |
| Sync mode (all streams) | **Incremental \| Append + Deduped** |
| Primary keys | Explicitly set per stream |
| Schema change handling | Manual approval (I approve all propagations myself) |
| Streams selected | 255 tables selected (not the full catalog) |

## Steps to reproduce

1. On a Postgres source configured with **Detect changes with Xmin System Column**, set up a connection to Redshift with a subset of tables selected.
2. Set all streams to **Incremental | Append + Deduped** and define the primary key(s) for each stream. In the schema tab, all streams show `` for the cursor field (no cursor is manually set, as expected for Xmin).
3. Set schema change propagation to require manual approval.
4. Upgrade the **source-postgres** connector from **3.7.2** to **3.8.0** (or **3.8.1**).
5. Trigger/receive a source schema change. In this case the change is non-breaking — a single new column added to a stream that is already being synced.
6. Open the connection → **Schema** tab → **Review changes**.

## Expected behavior

- The change review popup shows only the non-breaking change (one new column on one existing stream).
- All Xmin-based streams continue to show `` for the cursor field.
- The changes can be saved without any additional manual cursor selection.

## Actual behavior

- The review popup correctly shows only the single new column being added.
- However, **behind the popup, 4 streams in the schema list show "Missing cursor" (in red)** instead of ``.
- Airbyte **refuses to save** the changes until a cursor column is manually selected for each of those 4 streams via the dropdown in the cursor error state.
- Only **4 of the 255** selected tables are affected.

## Reproducibility / scope

- The failure is **deterministic**: downgrading to 3.7.2 and upgrading again reproduces the error on the **exact same 4 tables** every time.
- The **same 4 tables** are affected across **2 isolated environments**, so this is not specific to a single instance's state.
- No configuration on the connection or streams was changed other than the connector version.

## Affected tables

The 4 affected tables belong to a group of **append-only audit/history tables** that capture changes to other tables (populated by a CDC worker built on AWS DMS and custom workflows). From the schema perspective there does not appear to be anything that should affect Xmin usage. Representative DDL for one of them:

```sql
CREATE TABLE items_change_history (
pk_items_account_id VARCHAR NULL,
pk_items_category_id VARCHAR NULL,
pk_items_code VARCHAR NULL,
id UUID NOT NULL DEFAULT uuid_generate_v4(),
reason VARCHAR NULL,
initiator VARCHAR NULL,
trace VARCHAR NULL,
"timestamp" TIMESTAMP WITH TIME ZONE NOT NULL,
old_values JSONB NULL,
new_values JSONB NULL,

CONSTRAINT items_change_history_pkey
PRIMARY KEY (id),
CONSTRAINT items_change_history_pk_items_fkey
FOREIGN KEY (
pk_items_account_id,
pk_items_category_id,
pk_items_code
) REFERENCES items (account_id, category_id, code)
);
```

### Characteristics of the affected tables

- **Append-only** audit/history tables: rows are only ever `INSERT`ed, never `UPDATE`d or `DELETE`d.
- Populated by a **CDC worker** (AWS DMS + custom workflows), not by application code or triggers.
- Primary key is a **random UUID v4** (`uuid_generate_v4()`), i.e. **not time-ordered**.
- The only timestamp column is named **`"timestamp"`** — a PostgreSQL reserved word, hence quoted.
- **`REPLICA IDENTITY`** is left at **`DEFAULT`** (never set to `FULL`).
- The **parent table (`items`)** has `REPLICA IDENTITY FULL`; the history tables themselves do **not**.
- No triggers, no functions, no partitioning.
- No additional indexes beyond the implicit PK index on `id`.
- No `created_at` / `updated_at` columns.

## Notes / possible leads

Since Xmin is source-defined and does not rely on any user column, it is unclear why the connector would drop the source-defined cursor for these specific streams while leaving the other 251 intact. A few characteristics stand out as potential contributing factors worth checking:

- The presence of a column named `"timestamp"` (a reserved word) on all 4 affected tables.
- The random (non-monotonic) UUID primary key.
- The `REPLICA IDENTITY` configuration differing between parent and history tables.

Due to confidentiality reasons I cannot provide logs or exact tables definitions.

## Impact

This blocks routine, non-breaking schema change propagation: a single added column cannot be approved without manually assigning cursors to unrelated streams, which is both incorrect (these streams should use Xmin) and operationally disruptive for an otherwise automated pipeline.

### Relevant log output

```shell
None
```

### Contribute

- [ ] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13051

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。