No way to bound memory use when compacting tables with wide rows
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 118
Description
We run compactions against tables with fairly wide rows and found it hard to keep worker memory under control. Looking at the read+write pipeline, the knobs that matter are either hardcoded or only settable as table properties:
- the scanner decodes `read.parquet.batch-size` rows per Arrow batch (default 131072) and there is no per-scan override, so lowering it for a compaction means changing the table property for every reader
- each rolling data writer buffers up to 64 record batches in its input channel (`rollingDataWriterQueueCapacity` is a constant)
- `write.parquet.row-group-limit` is also table-wide only
Peak memory ends up being roughly decode batch size times buffered batches times row width. With the defaults and rows in the KB range that arithmetic gets into gigabytes per worker before the parquet writer even flushes anything.
I'd like to add per-scan and per-write overrides for these (a scan option for the decode batch size, a write option for the writer queue capacity and the row group limit) and forward them through `CompactionGroupOption` so `ExecuteCompactionGroup` can run with tighter bounds without touching table properties. Defaults would stay exactly as they are today.
I have a working patch with tests and would like to pick this up, happy to put up a PR if this sounds reasonable.
Contributor guide
Research direction
Start with the read+write pipeline, the rollingDataWriterQueueCapacity constant, CompactionGroupOption, and ExecuteCompactionGroup. Trace how scan and write options reach compaction, then run the existing tests mentioned in the issue. Done means per-scan and per-write overrides work, compaction forwards them, and current defaults remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100