Basekick-Labs / Basekick-Labs/arc

feat(ingestion): add backpressure when memory buffers exceed threshold

Open
#257 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

## Problem

Under sustained high-throughput ingestion (~1M+ rps), Arrow buffers accumulate data faster than they can flush to Parquet. This causes OOMKilled on memory-constrained instances (e.g., 1 vCPU / 2GB RAM with `ARC_DATABASE_MEMORY_LIMIT=1GB`).

Observed behavior:
- Flush durations increase from ~600ms to ~2600ms as buffer sizes grow
- Records per flush climb from 770K to 3.1M+
- Eventually the container is OOMKilled (exit code 137)

## Proposed Solution

Add ingestion backpressure at the write handler level:
- Track current Arrow buffer memory usage (the writer already knows this)
- When buffer memory exceeds a configurable threshold (e.g., 75% of `ARC_DATABASE_MEMORY_LIMIT`), reject writes with HTTP 429 and a `Retry-After` header
- When buffers drain below the threshold, resume accepting writes

This is a standard pattern in production databases — protect the process rather than accepting writes until OOM.

## Configuration

- `ARC_INGESTION_BACKPRESSURE_THRESHOLD` — percentage of memory limit (default: 75%)
- Response: `429 Too Many Requests` with `Retry-After: 1` header

## Context

Discovered during sustained load testing through TLS reverse proxy on a 1 vCPU / 2GB instance. 15 workers at 500 batch size sustained ~1M rps for 60s before OOM on subsequent larger tests.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.