apache / apache/arrow-rs-object-store

[object_store] API to calculate memory consumption of `ObjectMeta`

Open
#136 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In InfluxDB IOx we cache `ObjectMeta` (among other things) and thus calculate the memory used in order to limit the amount of memory used for caching

**Describe the solution you'd like**

In order to have accurate memory usage accounting, it would be nice to have an API like
```rust
impl ObjectMeta {
/// return the total bytes of memory used by this object, including heap space
pub fn memory_bytes(&self) -> usize {
...
}
}
```

**Describe alternatives you've considered**
We could cache the raw data directly as `Strings` and recreate `ObjectMeta` when needed

**Additional context**
See context in https://github.com/influxdata/influxdb_iox/pull/8577/files#r1316378812

Contributor guide

Open the contributing guide

Research direction

Start by locating the Rust definition of ObjectMeta and reviewing how its stored data contributes to memory use. Add the requested memory_bytes API, with completion defined as reporting the total bytes used by an ObjectMeta instance, including heap space.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.