airbytehq / airbytehq/airbyte

Dataplane Group Mismatch After Upgrade from 1.7.x to 2.0.x — All Syncs Stuck in "Queued" / "Workload Never Claimed"

Đang mở
#74,275 8 bình luận 3 reaction 0 người được giao Xem trên GitHub
area/platform autoteam community needs-triage team/compose team/platform-move type/bug
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.3k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

### Helm Chart Version

2.0.1 (upgraded from 1.7.7)

### What step the error happened?

Upgrading the Platform or Helm Chart

### Relevant information

fter upgrading from Airbyte 1.7.7 to 2.0.1 using `abctl`, all syncs were permanently stuck in **"Queued"** status with the error:

> `Airbyte could not start the process within time limit. The workload was never claimed.`

And no logs were available for any job:

> `No logs found for this job.`

### Reproduction Steps

1. Install Airbyte 1.7.7 using `abctl local install`
2. Run several syncs to build up workflow history in Temporal
3. Upgrade to 2.0.1: `abctl local install --chart-version 2.0.1`
4. Attempt to trigger any sync

### Root Cause

The bootloader creates a **new `dataplane_group`** on every fresh install/upgrade with a new UUID (e.g. `11bbbe03-17ca-4017-b7df-50d5469058eb`), while the existing database already contains an `AUTO` dataplane group from the previous installation (`56321339-3ce3-4a9d-8d78-9c4db27796a5`).

The workload-launcher registers itself under the **new** group, but new workloads are routed to the **old** `AUTO` group. Since the launcher only polls for workloads matching its own group ID, every workload sits in `pending` forever and is never claimed.

**Database evidence:**

```sql
-- Workload routed to OLD group
SELECT id, status, dataplane_group FROM workload WHERE status = 'pending';
-- Result: dataplane_group = 56321339-3ce3-4a9d-8d78-9c4db27796a5 (OLD AUTO group)

-- Launcher registered under NEW group
SELECT d.id, d.dataplane_group_id, dg.name
FROM dataplane d
JOIN dataplane_group dg ON dg.id = d.dataplane_group_id
WHERE d.tombstone = false;
-- Result: launcher dataplane under 11bbbe03-17ca-4017-b7df-50d5469058eb (NEW unnamed group)
```

**Workload-launcher logs confirming it never polls:**

The launcher only runs pod sweeper cycles with no `ApiQueuePoller` initialization:
```
INFO i.a.w.l.PodSweeper(sweepPods) - Starting pod sweeper cycle...
INFO i.a.w.l.PodSweeper(sweepPods) - Completed pod sweeper cycle.
```

Expected healthy state (missing after upgrade):
```
INFO i.a.w.l.p.c.WorkloadApiQueuePoller(resumePolling) - Resuming ApiQueuePoller with and high
INFO i.a.w.l.p.c.WorkloadApiQueuePoller(resumePolling) - Resuming ApiQueuePoller with and default
```

**Additional contributing factor:**

Stale Temporal `ConnectionManagerWorkflow` and `JobPostProcessingWorkflow` instances carried over from 1.7.x also caused `NonDeterministicException` errors in the worker on startup, which needed to be terminated separately via `tctl`.

### Workaround

After taking a database backup, apply the following fix to move all dataplanes to the existing `AUTO` group:

```sql
UPDATE dataplane
SET dataplane_group_id = ''
WHERE dataplane_group_id = '';
```

Then restart the workload-launcher:
```bash
kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
```

The launcher logs should then show:
```
INFO Resuming ApiQueuePoller with and high
INFO Resuming ApiQueuePoller with and default
```

**Note:** This fix needs to be reapplied after every `abctl local install` run as the bootloader recreates the mismatch on each startup.

### Expected Behavior

After upgrading from 1.7.x to 2.0.x, the bootloader should either:
- Reuse the existing `AUTO` dataplane group instead of creating a new one, OR
- Migrate existing workload routing to the newly created group

### Actual Behavior

The bootloader creates a new unnamed dataplane group on every install. The workload-launcher registers under this new group, but workloads continue to be routed to the old `AUTO` group, causing all syncs to time out with "workload was never claimed".

### Relevant log output

```shell
### Workload stuck in pending (never claimed)

SELECT id, status, termination_reason FROM workload WHERE id LIKE '%%';
-- termination_reason: "Airbyte could not start the process within time limit. The workload was never claimed."

### Worker NonDeterministicException (stale Temporal workflows from 1.7.x)

io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event of type 'EVENT_TYPE_WORKFLOW_TASK_STARTED' during execution.
...
Caused by: io.temporal.worker.NonDeterministicException

### Workload-launcher authentication failure (secondary issue triggered by manual intervention)

ERROR i.a.a.c.a.AccessTokenInterceptor(intercept) - Failed to obtain or add access token
java.lang.Exception: Token request failed with HTTP code 500
...
Caused by: ClientException: Client error : 401 Unauthorized
ERROR i.m.r.Micronaut(handleStartupException) - Error starting Micronaut server: Failed to initialize data-plane
java.lang.RuntimeException: Failed to initialize data-plane
```

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11516

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.