airbytehq / airbytehq/airbyte

[source-shopify] Bulk streams silently drop the whole result and advance state when a bulk job finishes with objectCount >= job_checkpoint_interval

Abierto
#85,372 0 comentarios 0 reacciones 0 asignados Ver en GitHub
area/connectors autoteam community connectors/source/shopify needs-triage team/extensibility type/bug
Lenguaje dominante
Python
Estrellas
22.1k
Forks
5.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

### Connector Name

source-shopify

### Connector Version

3.5.1 (platform 2.0.1, self-hosted on GKE Autopilot). The affected code path is unchanged on `master` (4.0.0), see below.

### What step the error happened?

During the sync

## Summary

Incremental GraphQL Bulk streams lose the complete result of a bulk job whenever Shopify's `objectCount` for that job is greater than or equal to the configured `job_checkpoint_interval`. The connector logs `checkpointing after >= N rows collected`, then `Rows collected: X --> records: 0` and `continue from checkpoint: None`, advances the stream state past the slice and continues with the next slice. The sync ends with status **succeeded**. No warning, no failure, no retry: the records of that slice are silently lost, and because no records are emitted the cursor never advances, so every following sync repeats the oversized job and the stream never recovers on its own.

## Evidence

We collected the `Rows collected: X --> records: Y` log lines of 2,353 bulk jobs across three Shopify sources (DE with `job_checkpoint_interval = 1,000,000`, CH with `400,000`, partner with `100,000`) between 2026-08-15 and 2026-09-07:

| Source | `job_checkpoint_interval` | Largest job that produced records | Smallest job with 0 records |
|---|---|---|---|
| CH | 400,000 | 267,558 rows → 267,540 records | 657,409 rows → 0 records |
| DE | 1,000,000 | 839,146 rows → 825,156 records | 1,038,235 rows → 0 records |

- 75 of 75 bulk jobs at or above the interval produced 0 new records.
- All 2,278 bulk jobs below the interval produced records as expected (the only 0-record jobs below the interval are parent-only results without child records, e.g. `inventory_levels` with one row).

Affected streams in our stores: `discount_codes` (every run since 2026-08-23 in both stores, 0.7–2.7M rows per run) and `metafield_customers` (a single-day slice with 4.47M rows after a bulk customer update). The `discount_codes` cursor has been stuck since 2026-08-20 in both stores; the window and therefore the row count grew with every run, and every run showed "succeeded".

## Log excerpt

Stream `metafield_customers`, `bulk_window_in_days = 1`, `job_checkpoint_interval = 1000000`:

```
Stream: `metafield_customers` requesting BULK Job for period: 2026-08-27T02:41:48+00:00 -- 2026-08-28T02:41:48+00:00. Slice size: `P1D`. The BULK checkpoint after `1000000` lines.
Stream: `metafield_customers`, the BULK Job: `gid://shopify/BulkOperation/12015924347229` is CREATED
... (job reported as CREATED for ~9 minutes)
Stream: `metafield_customers`, checkpointing after >= `1000000` rows collected.
Stream: `metafield_customers`, the BULK Job: `gid://shopify/BulkOperation/12015924347229` time elapsed: 567.161 sec. Rows collected: 4466586 --> records: `0`.
Stream metafield_customers, continue from checkpoint: `None`.
Stream: `metafield_customers` requesting BULK Job for period: 2026-08-28T02:41:48+00:00 -- 2026-08-29T02:41:48+00:00. Slice size: `P1D`. ...
Stream: `metafield_customers`, the BULK Job: `gid://shopify/BulkOperation/12015976808797` time elapsed: 24.784 sec. Rows collected: 45768 --> records: `43296`.
```

The same three lines appear for `discount_codes` on a job that was polled while RUNNING (two RUNNING polls; the last one saw 84,088 rows, the next poll found the job finished with 2,677,764 rows):

```
Stream: `discount_codes`, the BULK Job: `gid://shopify/BulkOperation/12009169682781` is RUNNING. Elapsed time: 627 sec. Rows collected: 84088.
Stream: `discount_codes`, checkpointing after >= `1000000` rows collected.
Stream: `discount_codes`, the BULK Job: `gid://shopify/BulkOperation/12009169682781` time elapsed: 2489.196 sec. Rows collected: 2677764 --> records: `0`.
Stream discount_codes, continue from checkpoint: `None`.
Read 0 records from discount_codes stream
```

No warning or error is logged; the sync finishes with `succeeded`.

## Suspected mechanism

`ShopifyBulkManager` polls the bulk operation every `job_check_interval`. When a poll observes `objectCount >= job_checkpoint_interval`, `_on_running_job` calls `_cancel_on_checkpointing()` → `_job_cancel()`. If Shopify has finished the operation in the meantime (large jobs finish between two polls, or the job goes from CREATED straight to done), the cancel has nothing to cancel, `_on_canceled_job` → `_job_get_checkpointed_result()` finds no partial result to download, the record producer composes 0 records, the checkpoint cursor is `None`, and the slice is treated as done. Nothing in this path raises, so the platform marks the sync successful.

This code path is unchanged on `master` (4.0.0): `_on_running_job` → `_cancel_on_checkpointing` → `_on_canceled_job` → `_job_get_checkpointed_result`. Upgrading the connector does not help; the 3.5.2 and 4.0.0 releases only touch shop-config validation and the `processed_at` schema annotations.

## Expected behavior

Either the finished result is downloaded and processed in full when the job is already COMPLETED (the checkpoint threshold should only cancel a job that is still running), or the slice fails loudly / is re-sliced so that the sync never reports success while dropping the data.

## Impact

Silent data loss in incremental syncs, not detectable from the Airbyte UI or job status. In our case `discount_codes` stopped delivering data in both stores for two weeks, and a bulk customer update day cannot be synced for `metafield_customers` at all, while every sync showed "succeeded".

## Workarounds we applied

- `bulk_window_in_days = 1` to keep slices below the threshold on normal days.
- `job_checkpoint_interval = 1,000,000` (the maximum) on all sources.
- Switched `discount_codes` to the new `discount_codes_sync` stream (works).
- A single day with a bulk customer update (330k customers, 4.47M metafield rows) still cannot be synced with the bulk stream.

## Related

- #78512 (`discount_codes` nested `codes` truncation), same store, same team.
- The docs section "BULK job checkpoint collisions" describes a loud failure (`checkpoint collision is detected`); the behaviour above is the silent variant.

### Relevant log output

```shell

```

### Contribute

- [ ] Yes, I want to contribute

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.