Memory use trait / methods / macros
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
In #19599 we were discussing adding a `heap_size()` method to `Statistics`, which requires adding a similar method to all of the structs it references. The idea occurred that there are already crates to handle this sort of thing, namely:
- Our own [HeapSize](https://github.com/apache/arrow-rs/blob/b93fa52e47a83dff130709a6e80a4be5017dbf09/parquet/src/file/metadata/memory.rs#L33-L42) trait in `arrow-rs`
- [deepsize](https://crates.io/crates/deepsize) which is now unmaintained and it's [maintained but less widely used fork](https://github.com/nhtyy/deepsize2)
It seems that if we're going to be implementing this for multiple structs it would be nice to at least have a derivable trait and use that trait everywhere.
One choice is between using a 3rd party crate or our own trait. Another choice is what the semantics we want are, the most confusing one perhaps being ref counted pointers (`Arc`, `Rc`, etc.).
Having thought about it a bit I think we should go with `HeapSize` and buff that trait up to provide a macro, implement it for the standard library container types, etc.
That said I'm opening the issue to collect community feedback if anyone has opinions.
Contributor guide
Research direction
Start by reading issue #19599 and the existing HeapSize trait in arrow-rs, then compare the deepsize and deepsize2 approaches. Clarify whether DataFusion should use its own trait or a third-party crate, including semantics for Arc and Rc. Done means the community has agreed on the trait design and implementation scope; no DataFusion file or test is named here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100