GlobalMemoryManager negative accounting disables backpressure, causes OOM and stuck syncs
- Ngôn ngữ chính
- Python
- Star
- 22.1k
- Fork
- 5.3k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
## Bug Description
`GlobalMemoryManager.free()` allows `currentMemoryBytes` to go negative. Once negative, `requestMemory()` always grants new allocations (since `negative < maxMemoryBytes` is always true), disabling all buffer backpressure. This causes either:
1. **OOM crash** — unbounded buffering until the JVM runs out of heap (`java.lang.OutOfMemoryError: Java heap space`)
2. **Infinite zero-byte flush loop** — streams stuck flushing 0 records indefinitely while the memory counter sits at a deeply negative value
## Reproduction
This affects any destination using the async buffer framework (`FlushWorkers` / `BufferDequeue` / `GlobalMemoryManager`) when syncing large streams. We reproduced it with:
- **source-github** → **destination-redshift** (self-hosted OSS v2.0.1)
- The `pull_request_commits` stream on a large repo drives the counter to -1.5GB, then OOMs
Logs showing the negative accounting:
```
[ASYNC QUEUE INFO] Global: max: 1.5 GB, allocated: -1563831048 bytes (-1491 MB), %% used: -0.97
```
## Root Cause
**`BufferDequeue.take()`** frees `allocatedBytes - batchSizeBytes` when a queue is emptied. `allocatedBytes` comes from `queue.maxMemoryUsage`, which can drift higher than what `GlobalMemoryManager` actually granted — because `requestMemory()` returns 0 when memory is full, but the queue's internal counter is adjusted regardless. This results in freeing a negative value (or more than was allocated), driving `currentMemoryBytes` below zero.
**`GlobalAsyncStateManager.freeBytes()`** similarly frees state message bytes that may exceed what was actually allocated via `requestMemory()`.
Once `currentMemoryBytes` is negative, the gate in `requestMemory()` (`currentMemoryBytes >= maxMemoryBytes`) never triggers, and all backpressure is lost.
## Proposed Fix
1. **`GlobalMemoryManager.free()`**: CAS-clamp `currentMemoryBytes` to 0 when it would go negative
2. **`BufferDequeue.take()`**: Guard the free call to skip when `unusedBytes <= 0`
3. Two regression tests covering single and repeated over-free scenarios
PR: (will link after creation)
## Prior Issues
This is the same root cause as:
- #42109 — `[destination-bigquery] Infinite synchronisation - negativ memory allocation` (closed as stale, unfixed)
- #31905 — `[source-intercom] conversation_parts taking too long to sync // Freed more memory than allocated GCS` (closed)
- [Discussion #36827](https://github.com/airbytehq/airbyte/discussions/36827) — Mixpanel connector looping 0 bytes with negative memory
## Environment
- Airbyte OSS v2.0.1 (self-hosted via abctl)
- Java CDK `airbyte-cdk/java/airbyte-cdk/core`
- Affects all destinations using the async buffer framework
## Fix
PR: https://github.com/airbytehq/airbyte/pull/74898
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11670
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.