airbytehq / airbytehq/airbyte

source-shopify 3.5.1: source container OOMs (160MiB → >2GiB in <8 min) at end of `products` stream BULK walk — distinct from #76969 sort fix

Open
#81,518 3 comments 0 reactions 0 assignees View on GitHub
autoteam community connectors/destination/bigquery connectors/source/shopify team/use
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

### Connector Name

source-shopify

### Connector Version

3.5.1

### What step the error happened?

During the sync

### Relevant information

**Summary:** The source container is OOMKilled by a fast, transient memory spike (~160MiB → >2GiB in under 8 minutes) during/immediately after the `products` stream's GraphQL BULK historical walk. Steady-state memory for the rest of the sync is ~100–160MiB. This is a different code path than the OOM fixed in 3.3.2 (#76969, disk-backed external sort in `sort_output_asc()`) — that fix is already included in 3.5.1 and our spike occurs during bulk record production, not post-read sorting.

**Environment**
- source-shopify 3.5.1 (`airbyte/source-shopify:3.5.1`)
- destination-bigquery 3.0.19, container-orchestrator 2.0.1
- Airbyte OSS, Helm chart v1, Kubernetes (DigitalOcean DOKS)
- Connection-level `resource_requirements`: `memory_request 1Gi / memory_limit 2Gi` (per container)

**Sync shape**
- 7 streams: customers, metafield_products, order_refunds, orders, products, +2 more
- Full historical sync (state reset after upgrading from the declarative-manifest connector), BULK slices of `P30D` walking from 2022 to present
- Most BULK jobs complete sub-second; slices are mostly small/empty (many checkpoints with recordCount=0)

**Timeline of the failure (UTC)**
```
18:26:26 sync running; BULK walk proceeding normally (metafield_products etc.)
18:26–18:45 streams customers, metafield_products, order_refunds, orders complete;
products BULK walk proceeding. Source container memory sampled every
20s via cgroup: 102–159MiB the entire time. CPU < 5% of a core.
18:45:02 products requests its FINAL slice:
"Stream: `products` requesting BULK Job for period: 2026-06-28 ..."
BULK Job CREATED → COMPLETED (sub-second)
18:45:03 last log line relayed by orchestrator (checkpoint flush). Total log
silence from here on.
18:49 last cgroup sample before our sampler window ended: source at 158MiB
18:53:49 source container OOMKilled (exit 137) at the 2GiB cgroup limit
```
So the spike from ~160MiB to >2GiB happened in ≤8 minutes with no log output, right at/after the final `products` BULK slice completed.

**Why we believe this is a distinct, untracked bug**
- The JSONL bulk download/parse path in current master streams correctly (`request_kwargs={"stream": True}` + `iter_content` to disk in `bulk/job.py`; line-by-line `for line in jsonl_file:` in `bulk/record.py`) — so this is not whole-file buffering.
- #76969 (shipped 3.3.2) fixed the in-memory `sorted()` over the full emitted-record iterable; that fix is present in 3.5.1 and our profile doesn't match it (that one grew during record emission with logs flowing; ours spikes silently at a slice boundary).
- A remaining candidate from reading `bulk/record.py`: the per-parent record-components buffer (top-level record + all `__parentId`-linked children held until the next top-level record boundary). `products` is the most deeply nested bulk query (variants, images, media, metafields), so a query shape/ordering edge case that defeats the new-record boundary detection would produce exactly this unbounded growth. We have not confirmed this hypothesis — offered as a starting point.

**Workaround:** raising connection-level `memory_limit` to 4Gi lets the sync proceed (memory_request unchanged). Note we had to apply this via the internal config API / connection-level resource_requirements because of the known Helm v1 regression where `JOB_MAIN_CONTAINER_*` env vars are ignored for replication pods (#68162, #30814).

### Relevant log output

```
2026-07-08 18:45:02,913 ... Stream: `products`, the BULK Job: `gid://shopify/BulkOperation/5432867094621` time elapsed: ...
2026-07-08 18:45:02,913 ... Stream: `products` requesting BULK Job for period: 2026-06-28T00:00:00+00:00 -- ...
2026-07-08 18:45:02,913 ... Stream: `products`, the BULK Job: `gid://shopify/BulkOperation/5432867127389` is CREATED
2026-07-08 18:45:02,913 ... Stream: `products`, the BULK Job: `gid://shopify/BulkOperation/5432867127389` is COMPLETED
2026-07-08 18:45:03,018 ... CheckpointManager(flushStreamCheckpoint) [last line before silence]

(no further output; source container OOMKilled exit 137 at 18:53:49)
```

Kubernetes container status at failure:
```
source: reason=OOMKilled exit=137 finished=2026-07-08T18:53:49Z
(orchestrator and destination containers unaffected)
```

Sampled source-container memory (cgroup `memory.current`, 20s interval, 18:26–18:49): min 102MiB, max 159MiB — the entire spike to >2GiB occurred in the final ~5–8 minutes with no log output.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13076

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.