apache / apache/iceberg-rust

Unbounded object_cache size

Open
#1,720 1 comment 1 reaction 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 1h
Merged PRs (30d)
95

Description

### Apache Iceberg Rust version

None

### Describe the bug

Currently we use a weighed moka Cache to store read objects:

https://github.com/apache/iceberg-rust/blob/dc349284a4204c1a56af47fb3177ace6f9e899a0/crates/iceberg/src/io/object_cache.rs#L63-L69

However, the current implementation only takes the stack size of the struct itself into account.
Thus, a cached `ManifestList` would return a low `size_of_val` number, while its many `entries` could allocate a significant number of storage on the heap.

I am unsure if exact calculation is feasable. I would opt for a rough estimate, such as 2KB for a list and 8KB for a Manfest.

Let me know what you think!

### To Reproduce

Create a ManifestList with many entries. Read it cached, watch the memory usage significantly overshoot the set max_capacity.

### Expected behavior

_No response_

### Willingness to contribute

None

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/io/object_cache.rs at the linked cache configuration around lines 63–69, then reproduce the issue with a ManifestList containing many entries and observe memory usage against max_capacity. Determine a rough size estimate for cached ManifestList and Manifest objects, and consider how to verify that the cache no longer substantially overshoots its configured capacity.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.