apache / apache/arrow-rs

Parquet: Should DataPageV2 shrink_to_fit?

Open
#10,448 13 comments 0 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Is your feature request related to a problem or challenge?

There're two related changes:
1. DataPageV1 will shrink to fit if possible
2. https://github.com/apache/arrow-rs/pull/10345 This will avoid copying, but making more memory used

So, for DataPageV2, this would occupies more memory. Don't know should we enlarge memory?

### Describe the solution you'd like

IDK, maybe provide a shrink_to_fit ( which introduing memcpy ).

### Describe alternatives you've considered

IDK

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by comparing the memory behavior of Parquet DataPageV1 and DataPageV2, then read the linked pull request #10345 to understand the copy-avoidance change. The issue does not name files or tests; completion requires an agreed decision on whether DataPageV2 should expose or use shrink_to_fit and how its memory tradeoff should be validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.