airbytehq / airbytehq/airbyte

Dataplane Group Mismatch After Upgrade from 1.7.x to 2.0.x — All Syncs Stuck in "Queued" / "Workload Never Claimed"

未關閉
#74,275 8 則留言 3 個 reaction 已指派 0 人 在 GitHub 檢視
area/platform autoteam community needs-triage team/compose team/platform-move type/bug
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

### Helm Chart Version

2.0.1 (upgraded from 1.7.7)

### What step the error happened?

Upgrading the Platform or Helm Chart

### Relevant information

fter upgrading from Airbyte 1.7.7 to 2.0.1 using `abctl`, all syncs were permanently stuck in **"Queued"** status with the error:

> `Airbyte could not start the process within time limit. The workload was never claimed.`

And no logs were available for any job:

> `No logs found for this job.`

### Reproduction Steps

1. Install Airbyte 1.7.7 using `abctl local install`
2. Run several syncs to build up workflow history in Temporal
3. Upgrade to 2.0.1: `abctl local install --chart-version 2.0.1`
4. Attempt to trigger any sync

### Root Cause

The bootloader creates a **new `dataplane_group`** on every fresh install/upgrade with a new UUID (e.g. `11bbbe03-17ca-4017-b7df-50d5469058eb`), while the existing database already contains an `AUTO` dataplane group from the previous installation (`56321339-3ce3-4a9d-8d78-9c4db27796a5`).

The workload-launcher registers itself under the **new** group, but new workloads are routed to the **old** `AUTO` group. Since the launcher only polls for workloads matching its own group ID, every workload sits in `pending` forever and is never claimed.

**Database evidence:**

```sql
-- Workload routed to OLD group
SELECT id, status, dataplane_group FROM workload WHERE status = 'pending';
-- Result: dataplane_group = 56321339-3ce3-4a9d-8d78-9c4db27796a5 (OLD AUTO group)

-- Launcher registered under NEW group
SELECT d.id, d.dataplane_group_id, dg.name
FROM dataplane d
JOIN dataplane_group dg ON dg.id = d.dataplane_group_id
WHERE d.tombstone = false;
-- Result: launcher dataplane under 11bbbe03-17ca-4017-b7df-50d5469058eb (NEW unnamed group)
```

**Workload-launcher logs confirming it never polls:**

The launcher only runs pod sweeper cycles with no `ApiQueuePoller` initialization:
```
INFO i.a.w.l.PodSweeper(sweepPods) - Starting pod sweeper cycle...
INFO i.a.w.l.PodSweeper(sweepPods) - Completed pod sweeper cycle.
```

Expected healthy state (missing after upgrade):
```
INFO i.a.w.l.p.c.WorkloadApiQueuePoller(resumePolling) - Resuming ApiQueuePoller with and high
INFO i.a.w.l.p.c.WorkloadApiQueuePoller(resumePolling) - Resuming ApiQueuePoller with and default
```

**Additional contributing factor:**

Stale Temporal `ConnectionManagerWorkflow` and `JobPostProcessingWorkflow` instances carried over from 1.7.x also caused `NonDeterministicException` errors in the worker on startup, which needed to be terminated separately via `tctl`.

### Workaround

After taking a database backup, apply the following fix to move all dataplanes to the existing `AUTO` group:

```sql
UPDATE dataplane
SET dataplane_group_id = ''
WHERE dataplane_group_id = '';
```

Then restart the workload-launcher:
```bash
kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
```

The launcher logs should then show:
```
INFO Resuming ApiQueuePoller with and high
INFO Resuming ApiQueuePoller with and default
```

**Note:** This fix needs to be reapplied after every `abctl local install` run as the bootloader recreates the mismatch on each startup.

### Expected Behavior

After upgrading from 1.7.x to 2.0.x, the bootloader should either:
- Reuse the existing `AUTO` dataplane group instead of creating a new one, OR
- Migrate existing workload routing to the newly created group

### Actual Behavior

The bootloader creates a new unnamed dataplane group on every install. The workload-launcher registers under this new group, but workloads continue to be routed to the old `AUTO` group, causing all syncs to time out with "workload was never claimed".

### Relevant log output

```shell
### Workload stuck in pending (never claimed)

SELECT id, status, termination_reason FROM workload WHERE id LIKE '%%';
-- termination_reason: "Airbyte could not start the process within time limit. The workload was never claimed."

### Worker NonDeterministicException (stale Temporal workflows from 1.7.x)

io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event of type 'EVENT_TYPE_WORKFLOW_TASK_STARTED' during execution.
...
Caused by: io.temporal.worker.NonDeterministicException

### Workload-launcher authentication failure (secondary issue triggered by manual intervention)

ERROR i.a.a.c.a.AccessTokenInterceptor(intercept) - Failed to obtain or add access token
java.lang.Exception: Token request failed with HTTP code 500
...
Caused by: ClientException: Client error : 401 Unauthorized
ERROR i.m.r.Micronaut(handleStartupException) - Error starting Micronaut server: Failed to initialize data-plane
java.lang.RuntimeException: Failed to initialize data-plane
```

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11516

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。