Sync job marked succeeded with 0 records loaded when its only attempt dies early (no retry scheduled)
- Lenguaje dominante
- Python
- Estrellas
- 22.1k
- Forks
- 5.3k
- Métricas de merge de PR
- Métricas de PR pendientes
Descripción
## Topic
Platform / job orchestration: a sync job is reported as **"Sync succeeded — 0 bytes | no records loaded"** when its only attempt dies partway through reading, and no retry attempt is scheduled.
## Environment
- Airbyte OSS **1.8.2** (helm chart 1.8.2, self-hosted on GKE)
- Source: `source-mssql` 4.1.8 (custom image that only adds TLSv1 support for a legacy SQL Server; connector logic unchanged)
- Destination: `destination-bigquery` (Destinations V2 / typing+dedup)
- Connection: 10 streams, all **Full Refresh | Overwrite**, triggered externally via API once per day. The largest stream is ~102M records / ~280 GB and has **no source-side primary key**.
## Observed behavior
On some nights, the initial read attempt of the large stream dies partway. When that happens **early** (~20–50 min in), no second attempt is created and the job is marked **succeeded** with 0 records loaded:
| night | attempt | records emitted | records committed | outcome |
|---|---|---|---|---|
| A | #1 01:06–04:01 | 63.4M | 0 | attempt died |
| A | #2 04:02–08:37 | 101.8M | 101.8M | completed → job succeeded, 280 GB loaded ✅ |
| B | #1 01:04–01:54 | 24.3M | 0 | attempt died → **no retry → job "succeeded", 0 bytes** ❌ |
| C | #1 01:08–04:01 | 64.4M | 0 | attempt died |
| C | #2 04:02–08:38 | 101.9M | 101.9M | completed → job succeeded ✅ |
| D | #1 01:05–01:26 | 10.7M | 0 | attempt died → **no retry → job "succeeded", 0 bytes (35 min total)** ❌ |
(Numbers taken from `stream_stats` / `stream_statuses` / `jobs` in the internal DB.)
For the false-success jobs (B, D):
- `stream_stats` shows millions of `records_emitted` with `records_committed = 0`
- `stream_statuses` for the stream stays in `RUNNING` — it never transitions to `COMPLETE` or `INCOMPLETE`
- `jobs.status` still ends up `succeeded`, and the UI shows "Sync succeeded — 0 bytes | no records loaded"
- The final table correctly retains the previous generation (so no wrong data is served), but the day's sync is silently skipped
When the same attempt failure happens ~3h in (nights A and C), a second attempt **is** created and completes normally, so retry scheduling works in that path.
## Expected behavior
A job whose stream emitted records but never reached a terminal stream status, with nothing committed, should be marked **failed** (or retried), not `succeeded`. The false success defeats status-based monitoring/alerting — the failure is only detectable by inspecting per-stream stats.
## Notes
- The underlying attempt deaths are likely environmental on our side (long-running reads over a legacy TLSv1 connection; the ~3h deaths look like a session limit in our network path). This report is about the **job being reported as succeeded** in the early-death case, not about the attempt failure itself.
- Happy to provide worker/orchestrator logs for the affected jobs if useful.
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13372
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.