Parquet: Should DataPageV2 shrink_to_fit?
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Is your feature request related to a problem or challenge?
There're two related changes:
1. DataPageV1 will shrink to fit if possible
2. https://github.com/apache/arrow-rs/pull/10345 This will avoid copying, but making more memory used
So, for DataPageV2, this would occupies more memory. Don't know should we enlarge memory?
### Describe the solution you'd like
IDK, maybe provide a shrink_to_fit ( which introduing memcpy ).
### Describe alternatives you've considered
IDK
### Additional context
_No response_
Contributor guide
Research direction
Start by comparing the memory behavior of Parquet DataPageV1 and DataPageV2, then read the linked pull request #10345 to understand the copy-avoidance change. The issue does not name files or tests; completion requires an agreed decision on whether DataPageV2 should expose or use shrink_to_fit and how its memory tradeoff should be validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100