Reduce the `clone()` cost of `Statistics` by caching
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Wrapping the Statistics with `Arc<>'`s can be a solution?
Some structs like `FileGroup`, `PartitionedData` etc. caches the Statistics. So, if the source operators can access those, they should return over them. However, for other intermediate operators, perhaps we can utilize `PlanProperties`? The Statistics will be initiated once and cached like other planning properties
_Originally posted by @berkaysynnada in https://github.com/apache/datafusion/pull/15503#discussion_r2030231992_
Contributor guide
Research direction
No files or tests are named. Start by tracing where Statistics is cloned and how source operators, FileGroup, PartitionedData, and PlanProperties currently cache related state. Done means Statistics reuse is defined for the relevant operators and the clone cost is reduced without changing planning behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100