airbytehq / airbytehq/airbyte

GlobalMemoryManager negative accounting disables backpressure, causes OOM and stuck syncs

Aberta
#74,897 3 comentários 0 reações 0 responsáveis Ver no GitHub
autoteam community connectors/destination/redshift connectors/source/github team/use
Linguagem predominante
Python
Estrelas
22.1k
Forks
5.4k
Merge médio
5h
PRs com merge (30d)
671

Descrição

## Bug Description

`GlobalMemoryManager.free()` allows `currentMemoryBytes` to go negative. Once negative, `requestMemory()` always grants new allocations (since `negative < maxMemoryBytes` is always true), disabling all buffer backpressure. This causes either:

1. **OOM crash** — unbounded buffering until the JVM runs out of heap (`java.lang.OutOfMemoryError: Java heap space`)
2. **Infinite zero-byte flush loop** — streams stuck flushing 0 records indefinitely while the memory counter sits at a deeply negative value

## Reproduction

This affects any destination using the async buffer framework (`FlushWorkers` / `BufferDequeue` / `GlobalMemoryManager`) when syncing large streams. We reproduced it with:

- **source-github** → **destination-redshift** (self-hosted OSS v2.0.1)
- The `pull_request_commits` stream on a large repo drives the counter to -1.5GB, then OOMs

Logs showing the negative accounting:
```
[ASYNC QUEUE INFO] Global: max: 1.5 GB, allocated: -1563831048 bytes (-1491 MB), %% used: -0.97
```

## Root Cause

**`BufferDequeue.take()`** frees `allocatedBytes - batchSizeBytes` when a queue is emptied. `allocatedBytes` comes from `queue.maxMemoryUsage`, which can drift higher than what `GlobalMemoryManager` actually granted — because `requestMemory()` returns 0 when memory is full, but the queue's internal counter is adjusted regardless. This results in freeing a negative value (or more than was allocated), driving `currentMemoryBytes` below zero.

**`GlobalAsyncStateManager.freeBytes()`** similarly frees state message bytes that may exceed what was actually allocated via `requestMemory()`.

Once `currentMemoryBytes` is negative, the gate in `requestMemory()` (`currentMemoryBytes >= maxMemoryBytes`) never triggers, and all backpressure is lost.

## Proposed Fix

1. **`GlobalMemoryManager.free()`**: CAS-clamp `currentMemoryBytes` to 0 when it would go negative
2. **`BufferDequeue.take()`**: Guard the free call to skip when `unusedBytes <= 0`
3. Two regression tests covering single and repeated over-free scenarios

PR: (will link after creation)

## Prior Issues

This is the same root cause as:
- #42109 — `[destination-bigquery] Infinite synchronisation - negativ memory allocation` (closed as stale, unfixed)
- #31905 — `[source-intercom] conversation_parts taking too long to sync // Freed more memory than allocated GCS` (closed)
- [Discussion #36827](https://github.com/airbytehq/airbyte/discussions/36827) — Mixpanel connector looping 0 bytes with negative memory

## Environment

- Airbyte OSS v2.0.1 (self-hosted via abctl)
- Java CDK `airbyte-cdk/java/airbyte-cdk/core`
- Affects all destinations using the async buffer framework

## Fix

PR: https://github.com/airbytehq/airbyte/pull/74898

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11670

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.