airbytehq / airbytehq/airbyte

[helm] source-oracle OOM-killed on abctl/kind 2.1.0 — JDBC buffer estimator reads host memory (25GB) instead of the 2Gi container limit

Open
#82,129 3 comments 0 reactions 0 assignees View on GitHub
area/platform autoteam community connectors/destination/postgres connectors/source/oracle needs-triage team/compose team/platform-move type/bug
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

### Helm Chart Version

2.1.0

### What step the error happened?

During the Sync

### Relevant information

## Summary
On a self-managed abctl (kind) deployment, source-oracle reads are consistently OOM-killed on a small table (~72,466 rows / ~89 MB). The TwoStageSizeEstimator reports a max memory of ~25 GB (the host's total RAM) despite the source pod being launched with a 2Gi container limit. The JDBC buffer target is sized ~15 GB and the adaptive fetch size climbs to ~4.17 million rows, so the Oracle driver preallocates far beyond 2Gi and the pod is kernel-OOM-killed at ~70,000 records every run.

No documented memory control changes the reported max memory — it stays at ~25 GB regardless. This looks like the connector JVM not detecting the container cgroup memory limit in a kind/nested-container environment.

Possibly related to (but distinct from) the older frozen issue #27846 — this is specifically the TwoStageSizeEstimator reading host RAM instead of the 2Gi cgroup limit under abctl/kind, not a gradual leak.

## Environment
- abctl (kind single-node), Airbyte App/Chart version 2.1.0
- Host: OCI VM, ~25 GB RAM
- source-oracle: reproduced on both 0.5.7 and 0.5.8 (identical failure)
- destination-postgres 3.0.13, orchestrator/workload-init 2.1.0
- Sync mode: Full refresh | Append; 1 stream, ~72,466 rows, ~89 MB, 46 columns

## Timeline
- DB in continuous use since ~Nov 2025 across earlier Airbyte version(s).
- Current kind cluster rebuilt 2026-05-17 onto platform 2.1.0, carrying the existing DB forward.
- OOM on this table appears after the move to 2.1.0; other connections on the same instance sync fine.

## What I already ruled out (all evidence-confirmed)
1. JOB_MAIN_CONTAINER_MEMORY_LIMIT set to 6Gi — present in workload-launcher env, but source pod still launches at 2Gi.
2. CONNECTOR_SPECIFIC_RESOURCE_DEFAULTS_ENABLED=false — confirmed live; source pod still 2Gi.
3. DB-level resource_requirements — null on actor_definition, actor, and connection rows.
4. defaultRowPrefetch in JDBC params — fetch size still climbs to ~4.17M rows (per-statement setFetchSize overrides it).
5. JOB_DEFAULT_ENV_JAVA_OPTS=-Xmx1500m on the launcher — applied and restarted; source JVM still reports ~25GB max memory.
6. Connector version 0.5.7 vs 0.5.8 — identical failure; changelog shows no read-path change between them.

## Questions
- Is there a supported way to force the source connector JVM to honor the container cgroup limit (or a hard -Xmx) on abctl/kind 2.1.0?
- Is the workload-launcher expected to ignore JOB_MAIN_CONTAINER_MEMORY_LIMIT / CONNECTOR_SPECIFIC_RESOURCE_DEFAULTS_ENABLED for connector pods in 2.1.0?
- Should TwoStageSizeEstimator fall back to the cgroup limit rather than Runtime.maxMemory() when containerized?
- Interim: this stream has a JOURNAL_DATE cursor — would switching to incremental avoid the large-fetch preallocation, or does the initial backfill hit the same OOM?

### Relevant log output

```shell
[source] image: airbyte/source-oracle:0.5.8 resources: limits={memory=2Gi, cpu=2}, requests={memory=1Gi, cpu=1}
TwoStageSizeEstimator(getTargetBufferByteSize) Max memory limit: 25232932864, JDBC buffer size: 15139759718
AdaptiveStreamingQueryConfig(accept) Set new fetch size: 4170733 rows
Records read: 70000 (82 MB)
No exit code file found after waiting 10s. Container may have been killed (OOM or other signal). Returning fallback exit code 1.
Source process exited with non-zero exit code 1
sync summary: recordsEmitted 72405, recordsCommitted 0, status failed, failureOrigin source

```

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13107

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.