Dataplane Group Mismatch After Upgrade from 1.7.x to 2.0.x — All Syncs Stuck in "Queued" / "Workload Never Claimed"
- Ngôn ngữ chính
- Python
- Star
- 22.1k
- Fork
- 5.3k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
### Helm Chart Version
2.0.1 (upgraded from 1.7.7)
### What step the error happened?
Upgrading the Platform or Helm Chart
### Relevant information
fter upgrading from Airbyte 1.7.7 to 2.0.1 using `abctl`, all syncs were permanently stuck in **"Queued"** status with the error:
> `Airbyte could not start the process within time limit. The workload was never claimed.`
And no logs were available for any job:
> `No logs found for this job.`
### Reproduction Steps
1. Install Airbyte 1.7.7 using `abctl local install`
2. Run several syncs to build up workflow history in Temporal
3. Upgrade to 2.0.1: `abctl local install --chart-version 2.0.1`
4. Attempt to trigger any sync
### Root Cause
The bootloader creates a **new `dataplane_group`** on every fresh install/upgrade with a new UUID (e.g. `11bbbe03-17ca-4017-b7df-50d5469058eb`), while the existing database already contains an `AUTO` dataplane group from the previous installation (`56321339-3ce3-4a9d-8d78-9c4db27796a5`).
The workload-launcher registers itself under the **new** group, but new workloads are routed to the **old** `AUTO` group. Since the launcher only polls for workloads matching its own group ID, every workload sits in `pending` forever and is never claimed.
**Database evidence:**
```sql
-- Workload routed to OLD group
SELECT id, status, dataplane_group FROM workload WHERE status = 'pending';
-- Result: dataplane_group = 56321339-3ce3-4a9d-8d78-9c4db27796a5 (OLD AUTO group)
-- Launcher registered under NEW group
SELECT d.id, d.dataplane_group_id, dg.name
FROM dataplane d
JOIN dataplane_group dg ON dg.id = d.dataplane_group_id
WHERE d.tombstone = false;
-- Result: launcher dataplane under 11bbbe03-17ca-4017-b7df-50d5469058eb (NEW unnamed group)
```
**Workload-launcher logs confirming it never polls:**
The launcher only runs pod sweeper cycles with no `ApiQueuePoller` initialization:
```
INFO i.a.w.l.PodSweeper(sweepPods) - Starting pod sweeper cycle...
INFO i.a.w.l.PodSweeper(sweepPods) - Completed pod sweeper cycle.
```
Expected healthy state (missing after upgrade):
```
INFO i.a.w.l.p.c.WorkloadApiQueuePoller(resumePolling) - Resuming ApiQueuePoller with and high
INFO i.a.w.l.p.c.WorkloadApiQueuePoller(resumePolling) - Resuming ApiQueuePoller with and default
```
**Additional contributing factor:**
Stale Temporal `ConnectionManagerWorkflow` and `JobPostProcessingWorkflow` instances carried over from 1.7.x also caused `NonDeterministicException` errors in the worker on startup, which needed to be terminated separately via `tctl`.
### Workaround
After taking a database backup, apply the following fix to move all dataplanes to the existing `AUTO` group:
```sql
UPDATE dataplane
SET dataplane_group_id = ''
WHERE dataplane_group_id = '';
```
Then restart the workload-launcher:
```bash
kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
```
The launcher logs should then show:
```
INFO Resuming ApiQueuePoller with and high
INFO Resuming ApiQueuePoller with and default
```
**Note:** This fix needs to be reapplied after every `abctl local install` run as the bootloader recreates the mismatch on each startup.
### Expected Behavior
After upgrading from 1.7.x to 2.0.x, the bootloader should either:
- Reuse the existing `AUTO` dataplane group instead of creating a new one, OR
- Migrate existing workload routing to the newly created group
### Actual Behavior
The bootloader creates a new unnamed dataplane group on every install. The workload-launcher registers under this new group, but workloads continue to be routed to the old `AUTO` group, causing all syncs to time out with "workload was never claimed".
### Relevant log output
```shell
### Workload stuck in pending (never claimed)
SELECT id, status, termination_reason FROM workload WHERE id LIKE '%%';
-- termination_reason: "Airbyte could not start the process within time limit. The workload was never claimed."
### Worker NonDeterministicException (stale Temporal workflows from 1.7.x)
io.temporal.internal.statemachines.InternalWorkflowTaskException: Failure handling event of type 'EVENT_TYPE_WORKFLOW_TASK_STARTED' during execution.
...
Caused by: io.temporal.worker.NonDeterministicException
### Workload-launcher authentication failure (secondary issue triggered by manual intervention)
ERROR i.a.a.c.a.AccessTokenInterceptor(intercept) - Failed to obtain or add access token
java.lang.Exception: Token request failed with HTTP code 500
...
Caused by: ClientException: Client error : 401 Unauthorized
ERROR i.m.r.Micronaut(handleStartupException) - Error starting Micronaut server: Failed to initialize data-plane
java.lang.RuntimeException: Failed to initialize data-plane
```
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11516
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.