airbytehq / airbytehq/airbyte

Sync job marked succeeded with 0 records loaded when its only attempt dies early (no retry scheduled)

Abierto
#84,961 4 comentarios 0 reacciones 0 asignados Ver en GitHub
autoteam community connectors/destination/bigquery connectors/source/mssql team/use
Lenguaje dominante
Python
Estrellas
22.1k
Forks
5.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

## Topic

Platform / job orchestration: a sync job is reported as **"Sync succeeded — 0 bytes | no records loaded"** when its only attempt dies partway through reading, and no retry attempt is scheduled.

## Environment

- Airbyte OSS **1.8.2** (helm chart 1.8.2, self-hosted on GKE)
- Source: `source-mssql` 4.1.8 (custom image that only adds TLSv1 support for a legacy SQL Server; connector logic unchanged)
- Destination: `destination-bigquery` (Destinations V2 / typing+dedup)
- Connection: 10 streams, all **Full Refresh | Overwrite**, triggered externally via API once per day. The largest stream is ~102M records / ~280 GB and has **no source-side primary key**.

## Observed behavior

On some nights, the initial read attempt of the large stream dies partway. When that happens **early** (~20–50 min in), no second attempt is created and the job is marked **succeeded** with 0 records loaded:

| night | attempt | records emitted | records committed | outcome |
|---|---|---|---|---|
| A | #1 01:06–04:01 | 63.4M | 0 | attempt died |
| A | #2 04:02–08:37 | 101.8M | 101.8M | completed → job succeeded, 280 GB loaded ✅ |
| B | #1 01:04–01:54 | 24.3M | 0 | attempt died → **no retry → job "succeeded", 0 bytes** ❌ |
| C | #1 01:08–04:01 | 64.4M | 0 | attempt died |
| C | #2 04:02–08:38 | 101.9M | 101.9M | completed → job succeeded ✅ |
| D | #1 01:05–01:26 | 10.7M | 0 | attempt died → **no retry → job "succeeded", 0 bytes (35 min total)** ❌ |

(Numbers taken from `stream_stats` / `stream_statuses` / `jobs` in the internal DB.)

For the false-success jobs (B, D):

- `stream_stats` shows millions of `records_emitted` with `records_committed = 0`
- `stream_statuses` for the stream stays in `RUNNING` — it never transitions to `COMPLETE` or `INCOMPLETE`
- `jobs.status` still ends up `succeeded`, and the UI shows "Sync succeeded — 0 bytes | no records loaded"
- The final table correctly retains the previous generation (so no wrong data is served), but the day's sync is silently skipped

When the same attempt failure happens ~3h in (nights A and C), a second attempt **is** created and completes normally, so retry scheduling works in that path.

## Expected behavior

A job whose stream emitted records but never reached a terminal stream status, with nothing committed, should be marked **failed** (or retried), not `succeeded`. The false success defeats status-based monitoring/alerting — the failure is only detectable by inspecting per-stream stats.

## Notes

- The underlying attempt deaths are likely environmental on our side (long-running reads over a legacy TLSv1 connection; the ~3h deaths look like a session limit in our network path). This report is about the **job being reported as succeeded** in the early-death case, not about the attempt failure itself.
- Happy to provide worker/orchestrator logs for the affected jobs if useful.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13372

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.