airbytehq / airbytehq/airbyte

Platform Issue: launch-side workload failures are recorded as successful syncs (workload.status=failure, jobs/attempts.status=succeeded)

Open
#85,355 0 comments 0 reactions 0 assignees View on GitHub
autoteam community team/use
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

### Platform Version

`2.1.0` — all platform images at `2.1.0` (`server`, `worker`, `workload-launcher`, `workload-api-server`, `cron`), deployed via `abctl` v0.30.4 (single-node kind cluster, chart V2).

### What step the error happened?

During the Sync

### Relevant information

**Summary:** when a sync's workload fails on the **launch** side — Kubernetes cannot schedule the replication pod and the launcher raises `ResourceConstraintException` — the platform correctly records the *workload* as failed, but records the corresponding **job and attempt as `succeeded`**, with a `NULL` `failure_summary` and no `sync_stats` / `stream_stats` rows.

The result is a sync that moves zero bytes and reports total success: green check in the UI, `status: succeeded` from the Public API, and a **"Sync Success" webhook notification with `success: true`**. There is also no retry, because as far as the job state machine is concerned nothing went wrong.

The failure information is not lost — it is written to the `workload` table and to the launcher log. It simply never propagates back into `jobs` / `attempts`.

#### Evidence

One sync, job `1530`. The workload row (connection UUID redacted):

```
id | _1530_0_sync
status | failure
type | sync
termination_source | launch
termination_reason | io.airbyte.workers.exception.ResourceConstraintException: Unable to start
the REPLICATION pod. This may be due to insufficient system resources.
Please check available resources and try again.
created_at | 2026-09-04 14:00:34+00
updated_at | 2026-09-04 14:15:35+00
```

The job and attempt rows for that same sync:

```
job_id | 1530
job_status | succeeded <--
config_type | sync
attempt_id | 1622
attempt_number | 0
attempt_status | succeeded <--
failure_summary | (null) <--
att_created | 2026-09-04 14:00:01+00
ended_at | 2026-09-04 14:15:35+00
sync_stats_rows | 0
```

The attempt "succeeded" in 15m34s having produced no `sync_stats` row at all. The same connection on a healthy day runs ~110 minutes and commits ~120k records.

#### This is not specific to `ResourceConstraintException`

Joining every failed sync workload on this instance back to its job and attempt (~3 months of history), whether the failure propagates depends entirely on `termination_source`:

| `termination_source` | `workload.status` | `jobs.status` | `attempts.status` | `failure_summary` | count |
|---|---|---|---|---|---|
| `source` | failure | failed | failed | set | 75 |
| `source` | failure | cancelled | failed | set | 42 |
| `source` | failure | succeeded¹ | failed | set | 9 |
| `destination` | failure | cancelled | failed | set | 3 |
| `destination` | failure | succeeded¹ | failed | set | 1 |
| **`launch`** | failure | **succeeded** | **succeeded** | **null** | **40** |
| **`workload-monitor-heartbeat`** | failure | **succeeded** | **succeeded** | **null** | **4** |
| **`airbyte_platform`** | failure | **succeeded** | **succeeded** | **null** | **3** |

¹ job `succeeded` with attempt `failed` is the normal retry case — an early attempt failed, a later one succeeded. Correct behavior.

Connector-side terminations propagate correctly in all 130 cases: the attempt is always marked `failed` and carries a failure summary. **Platform-side terminations never propagate — 47 of 47 were recorded as fully successful attempts.** The other two reasons observed were:

- `workload-monitor-heartbeat` — `Airbyte could not track the sync progress. Sync process exited without reporting status.`
- `airbyte_platform` — `Something went wrong within the airbyte platform`

Both of those are, if anything, even more clearly failures than the launch case.

#### Impact

This is silent data loss. On our instance one connection failed this way on **16 consecutive days** without a single visible error anywhere in Airbyte — it was eventually noticed only because a downstream BI dashboard stopped moving. Because the platform reports success:

- the UI, the Public API and `GET /jobs` all show a healthy connection;
- the failure notification never fires, and the **success** notification does;
- any downstream orchestration keyed off the sync-success webhook proceeds on stale data and also reports green;
- no retry is attempted.

An operator has no supported way to detect this. The only reliable signal we found is querying the internal metadata database for attempts that are `succeeded` but have no `sync_stats` row — which is not a stable public interface.

Note that `recordsCommitted == 0` is **not** a usable proxy: a legitimately empty sync also reports zero records, and the webhook payload for a phantom success (`recordsCommitted: 0, bytesCommitted: 0, success: true`) is byte-identical to that of a genuinely empty sync.

#### Steps to reproduce

1. Deploy Airbyte 2.1.0 on a node with limited allocatable CPU/memory.
2. Schedule several connections to start at the same time, enough that the pending replication pods exceed what the node can schedule.
3. Wait ~15 minutes for the launcher to give up.
4. The launcher logs `Stage Pipeline Exception: … ResourceConstraintException: Unable to start the REPLICATION pod`, `workload.status` becomes `failure` with `termination_source = launch` — and the sync appears in the UI as **Succeeded**.

#### Expected behavior

A workload that terminates with `status = failure` should mark its attempt (and, once retries are exhausted, its job) as `failed`, and populate `failure_summary` from `termination_reason`, regardless of whether the termination originated in the connector or in the platform. A sync that never started the replication pod must not be reported as a successful sync.

#### Version note

I have not been able to test against 2.1.1; its release notes list only the connection-cap fix, so I don't believe this is addressed there. Happy to gather more detail from this instance if useful.

### Relevant log output

```shell
2026-09-02 07:15:25 platform ERROR Stage Pipeline Exception: io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.ResourceConstraintException: Unable to start the REPLICATION pod. This may be due to insufficient system resources. Please check available resources and try again.
message: io.airbyte.workers.exception.ResourceConstraintException: Unable to start the REPLICATION pod. This may be due to insufficient system resources. Please check available resources and try again.
stackTrace: [Ljava.lang.StackTraceElement;@3a0e4056
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.