Add max_open_partitions in FanoutWriter to help manage memory
Open
not-stale
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
> We can also introduce `max_open_partitions` here in the future to help avoid OOM
_Originally posted by @CTTY in [#1735](https://github.com/apache/iceberg-rust/pull/1735/files#r2425208882)_
Contributor guide
Research direction
Start by locating FanoutWriter and reviewing the referenced discussion in pull request #1735. Determine how max_open_partitions should limit open partitions to help avoid out-of-memory conditions, and verify the behavior against the existing FanoutWriter flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100