airbytehq / airbytehq/airbyte

GlobalMemoryManager negative accounting disables backpressure, causes OOM and stuck syncs

Đang mở
#74,897 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
autoteam community connectors/destination/redshift connectors/source/github team/use
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.3k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

## Bug Description

`GlobalMemoryManager.free()` allows `currentMemoryBytes` to go negative. Once negative, `requestMemory()` always grants new allocations (since `negative < maxMemoryBytes` is always true), disabling all buffer backpressure. This causes either:

1. **OOM crash** — unbounded buffering until the JVM runs out of heap (`java.lang.OutOfMemoryError: Java heap space`)
2. **Infinite zero-byte flush loop** — streams stuck flushing 0 records indefinitely while the memory counter sits at a deeply negative value

## Reproduction

This affects any destination using the async buffer framework (`FlushWorkers` / `BufferDequeue` / `GlobalMemoryManager`) when syncing large streams. We reproduced it with:

- **source-github** → **destination-redshift** (self-hosted OSS v2.0.1)
- The `pull_request_commits` stream on a large repo drives the counter to -1.5GB, then OOMs

Logs showing the negative accounting:
```
[ASYNC QUEUE INFO] Global: max: 1.5 GB, allocated: -1563831048 bytes (-1491 MB), %% used: -0.97
```

## Root Cause

**`BufferDequeue.take()`** frees `allocatedBytes - batchSizeBytes` when a queue is emptied. `allocatedBytes` comes from `queue.maxMemoryUsage`, which can drift higher than what `GlobalMemoryManager` actually granted — because `requestMemory()` returns 0 when memory is full, but the queue's internal counter is adjusted regardless. This results in freeing a negative value (or more than was allocated), driving `currentMemoryBytes` below zero.

**`GlobalAsyncStateManager.freeBytes()`** similarly frees state message bytes that may exceed what was actually allocated via `requestMemory()`.

Once `currentMemoryBytes` is negative, the gate in `requestMemory()` (`currentMemoryBytes >= maxMemoryBytes`) never triggers, and all backpressure is lost.

## Proposed Fix

1. **`GlobalMemoryManager.free()`**: CAS-clamp `currentMemoryBytes` to 0 when it would go negative
2. **`BufferDequeue.take()`**: Guard the free call to skip when `unusedBytes <= 0`
3. Two regression tests covering single and repeated over-free scenarios

PR: (will link after creation)

## Prior Issues

This is the same root cause as:
- #42109 — `[destination-bigquery] Infinite synchronisation - negativ memory allocation` (closed as stale, unfixed)
- #31905 — `[source-intercom] conversation_parts taking too long to sync // Freed more memory than allocated GCS` (closed)
- [Discussion #36827](https://github.com/airbytehq/airbyte/discussions/36827) — Mixpanel connector looping 0 bytes with negative memory

## Environment

- Airbyte OSS v2.0.1 (self-hosted via abctl)
- Java CDK `airbyte-cdk/java/airbyte-cdk/core`
- Affects all destinations using the async buffer framework

## Fix

PR: https://github.com/airbytehq/airbyte/pull/74898

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11670

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.