Basekick-Labs / Basekick-Labs/arc
feat(ingestion): add backpressure when memory buffers exceed threshold
- Dominant language
- Go
- Stars
- 677
- Forks
- 53
- Avg merge
- 9h 14m
- Merged PRs (30d)
- 164
Description
## Problem
Under sustained high-throughput ingestion (~1M+ rps), Arrow buffers accumulate data faster than they can flush to Parquet. This causes OOMKilled on memory-constrained instances (e.g., 1 vCPU / 2GB RAM with `ARC_DATABASE_MEMORY_LIMIT=1GB`).
Observed behavior:
- Flush durations increase from ~600ms to ~2600ms as buffer sizes grow
- Records per flush climb from 770K to 3.1M+
- Eventually the container is OOMKilled (exit code 137)
## Proposed Solution
Add ingestion backpressure at the write handler level:
- Track current Arrow buffer memory usage (the writer already knows this)
- When buffer memory exceeds a configurable threshold (e.g., 75% of `ARC_DATABASE_MEMORY_LIMIT`), reject writes with HTTP 429 and a `Retry-After` header
- When buffers drain below the threshold, resume accepting writes
This is a standard pattern in production databases — protect the process rather than accepting writes until OOM.
## Configuration
- `ARC_INGESTION_BACKPRESSURE_THRESHOLD` — percentage of memory limit (default: 75%)
- Response: `429 Too Many Requests` with `Retry-After: 1` header
## Context
Discovered during sustained load testing through TLS reverse proxy on a 1 vCPU / 2GB instance. 15 workers at 500 batch size sustained ~1M rps for 60s before OOM on subsequent larger tests.
Contributor guide
Assessment
This issue has not been assessed yet.