Full "Clear data" never erases global CDC shared state when orphaned stream states exist — stale Postgres LSN is unrecoverable via reset
- 主要语言
- Python
- 星标
- 22.1k
- 派生
- 5.3k
- PR 合并指标
- PR 指标待抓取
描述
## What happened
A Postgres CDC connection started failing every sync with:
> Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency to prevent this from happening in the future.
Following the error message and [the docs](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs), we ran a full connection reset ("Clear data") covering **all streams in the catalog**. The reset job succeeded (destination data cleared, per-stream states nulled), but the very next sync failed with the exact same error. `POST /api/v1/state/get` confirmed the global `shared_state` still contained the old Debezium offset:
```json
{
"stateType": "global",
"globalState": {
"shared_state": {
"state": {
"[\"...\",{\"server\":\"...\"}]": "{\"lsn_proc\":29225093698096,...,\"lsn\":29225093698096,...}"
}
},
"streamStates": [
{ "streamDescriptor": { "name": "SomeRemovedTable", "namespace": "public" }, "streamState": { ... } },
{ "streamDescriptor": { "name": "AnotherRemovedTable", "namespace": "public" }, "streamState": { ... } }
]
}
}
```
Note the two remaining `streamStates`: they belong to streams that were **removed from the connection's catalog** some time ago (moved to a separate connection). Their state rows were never deleted.
## Root cause
In `EmptyAirbyteSource.getNullGlobalMessage` ([EmptyAirbyteSource.kt#L269-L283](https://github.com/airbytehq/airbyte-platform/blob/3befb078dafe76c899cab3a909ca47448debfca5/airbyte-container-orchestrator/src/main/kotlin/io/airbyte/container/orchestrator/worker/io/EmptyAirbyteSource.kt#L269-L283)), the shared state is erased only when **every stream state present in the stored global state** is being reset:
```kotlin
// If all the streams in the current state have been reset, we consider this to be a full reset, so
// reset the shared state as well
if (currentState.global.streamStates.size.toLong() ==
globalState.streamStates.stream().filter { it.streamState == null }.count()
) {
globalState.sharedState = null
} else {
logger.info { "This is a partial reset, the shared state will be preserved" }
globalState.sharedState = currentState.global.sharedState
}
```
`streamsToReset` is derived from the connection's **current catalog**, so a stream state that is orphaned (its stream was removed from the connection) can never be included in it. Once a CDC connection has at least one orphaned stream state, **every** reset — including a full "Clear data" of all streams — takes the "partial reset" branch and preserves the stale Debezium offset. The connection is permanently unrecoverable through the UI or the reset/clear APIs.
## Steps to reproduce
1. Create a Postgres CDC (global state) connection syncing streams A, B, C.
2. Sync at least once, then remove stream C from the connection. Its row in the `state` table remains.
3. Invalidate the saved offset (e.g. WAL rotated past the slot's `confirmed_flush_lsn`) → syncs fail with "Saved offset is before replication slot's confirmed lsn".
4. Run "Clear data" / `POST /api/v1/connections/reset` for the whole connection. The job succeeds.
5. Next sync fails with the same error; `state/get` shows `shared_state` untouched plus the orphaned stream state for C.
## Expected behavior
A reset covering all streams in the catalog should be treated as a full reset and erase the shared state. Orphaned stream states (descriptors no longer present in the configured catalog) should be dropped or nulled as part of any reset — they are unreachable by definition.
## Workaround
Overwrite the state directly:
```
POST /api/v1/state/create_or_update
{
"connectionId": "...",
"connectionState": {
"stateType": "global",
"connectionId": "...",
"globalState": { "shared_state": {}, "streamStates": [] }
}
}
```
This empties the offset and the next sync re-snapshots. It does not remove the orphaned rows (`create_or_update` upserts only), so subsequent resets remain broken until they are cleaned up.
## Environment
- Airbyte Self-Managed Community (OSS), Kubernetes (Helm)
- Source: Postgres (CDC) → Destination: Snowflake
Happy to open a PR — the minimal fix in `getNullGlobalMessage` is to treat stream states whose descriptors are absent from the configured catalog as resettable when deciding whether the shared state can be erased.
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13127
贡献指南
评估
这个 Issue 还没有评估数据。