airbytehq / airbytehq/airbyte

Source check_connection hangs after database migration from chart 0.524.0 to 2.0.18 (via 1.6.0 intermediate step)

Aperta
#80,921 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area/abctl autoteam community connectors/destination/bigquery connectors/source/postgres needs-triage team/use type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### What happened?

# Source check_connection hangs after database migration from chart 0.524.0 to 2.0.18 (via 1.6.0 intermediate step)

[ related to https://github.com/airbytehq/airbyte/issues/80770. but with intermediate v1.6.0 migration step included ]

## Description

We are migrating Airbyte from chart `0.524.0` (platform `0.64.1`, abctl `v0.14.1`) to chart `2.0.18` (platform `2.0.0`, abctl `v0.30.4`). The airbyte database was copied from a Postgres 13 instance to a Postgres 16 instance. Temporal databases are created fresh (empty) since temporal workflow histories are incompatible across major versions.

Following the documented requirement that pre-1.6.0 installations must first upgrade to at least 1.6.0 before migrating to chart V2, we implemented a two-step migration:

### Step 1: chart 0.524.0 -> 1.6.0 (using abctl v0.29.0)

This step succeeds. The bootloader runs all flyway migrations (80 config migrations to `v2.0.0.010`, 8 jobs migrations to `v1.1.0.005`). Platform version updates from `0.64.1` to `1.6.0`. After upgrading connector versions via the UI (source-postgres to 3.8.1, destination-bigquery to 3.0.19), all source and destination tests pass. Connections are functional.

### Step 2: chart 1.6.0 -> 2.0.18 (using abctl v0.30.4)

This step installs successfully. The bootloader runs 37 additional config migrations and 4 jobs migrations. Platform version updates to `2.0.0`. All pods reach Running/Ready state, including the workload-launcher which authenticates successfully with the server.

However, **source check_connection hangs for exactly 300 seconds and then returns 502** (upstream prematurely closed connection). The server receives the request but:

- Produces zero log output about the check request
- Never dispatches a workload to the workload-launcher
- No check/discover pods are ever created
- The workload-launcher is healthy and polling for work -- it simply never receives any

This is the same behaviour we reported in the original ticket, but we can now confirm:

1. **Clean database + chart 2.0.18 -> works.** Check pods are created within seconds, sources verify immediately.
2. **Migrated database + chart 1.6.0 -> works.** Sources and destinations verified after connector upgrade via UI.
3. **Migrated database + chart 2.0.18 -> hangs.** Even with connector versions matching the clean install (source-postgres 3.8.1, destination-bigquery 3.0.19).

We have captured full diagnostic dumps at each stage (helm values, pod state, logs from all pods including ingress, database table exports, and full pg_dumps). Key observations from comparing the clean and migrated databases:

- K8s `airbyte-auth-secrets` match the `service_accounts` table -- auth is not the issue
- Connector versions and `default_version_id` in `actor_definition` match the clean install
- The migrated database has a residual dataplane/service_account from the V1 chart (the V2 bootloader creates a second set)
- The migrated database retains the `airbyte_configs` table (not present in clean installs)
- The server logs show `Log4j API could not find a logging provider` after startup (not present in clean install logs) -- this may explain why the server produces no output when the check request hangs

We are currently stopped at chart 1.6.0 in production since everything works at that version. We would like to migrate to chart V2 but cannot until this issue is resolved.

## What did you expect to happen?

Source check_connection should dispatch a check workload and return results, as it does with a clean database on the same chart version.

## Versions

| Component | Before | After (step 1) | After (step 2) |
|-----------|--------|-----------------|-----------------|
| abctl | v0.14.1 | v0.29.0 | v0.30.4 |
| Helm chart | 0.524.0 | 1.6.0 | 2.0.18 |
| Platform | 0.64.1 | 1.6.0 | 2.0.0 |
| source-postgres | 3.6.18 | 3.8.1 (UI upgrade) | 3.8.1 |
| destination-bigquery | 2.8.9 | 3.0.19 (UI upgrade) | 3.0.19 |
| PostgreSQL | 16.13 (RDS) | 16.13 | 16.13 |

## Diagnostic data

Diagnostic dumps available on request (helm values, pod logs, ingress logs, bootloader logs, database table exports for auth/dataplane/actor/secrets tables, full pg_dumps at each migration stage).

### What did you expect to happen?

Following migration to v2.0.18 all source and destination checks are launched and complete without error.

Instead, the checks are never launched and the UI returns with 502 failure after exactly 5 minutes.

### Abctl Version

```console
$ abctl version
version: v0.30.4
```

### Docker Version

```console
$ docker version
Client:
Version: 25.0.5
API version: 1.44
Go version: go1.22.5
Git commit: 5dc9bcc
Built: Wed Aug 21 00:00:00 2024
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 25.0.6
API version: 1.44 (minimum version 1.24)
Go version: go1.22.5
Git commit: b08a51f
Built: Wed Aug 21 00:00:00 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.20
GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353
runc:
Version: 1.1.13
GitCommit: 58aa9203c123022138b22cf96540c284876a7910
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```

### OS Version

```console
# On Linux:
$ cat /etc/os-release
AME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.5.20240916"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2028-03-15"```

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12980

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.