apache / apache/arrow-rs

Check valid byte_range in parquet Column Chunk reading?

Open
#6,255 4 comments 0 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

See: https://github.com/apache/parquet-testing/pull/58#issuecomment-2290985490

When reading a corrupt file, currently, arrow-rs would have:

```rust
/// Returns the offset and length in bytes of the column chunk within the file
pub fn byte_range(&self) -> (u64, u64) {
let col_start = match self.dictionary_page_offset() {
Some(dictionary_page_offset) => dictionary_page_offset,
None => self.data_page_offset(),
};
let col_len = self.compressed_size();
assert!(
col_start >= 0 && col_len >= 0,
"column start and length should not be negative"
);
(col_start as u64, col_len as u64)
}
```

Would we better check the range here?

**Describe the solution you'd like**

Checking the range when building the group reader or in "byte_range()"

**Describe alternatives you've considered**

```rust
/// Returns the offset and length in bytes of the column chunk within the file
pub fn byte_range(&self) -> Result<(u64, u64)> {
let col_start = match self.dictionary_page_offset() {
Some(dictionary_page_offset) => dictionary_page_offset,
None => self.data_page_offset(),
};
let col_len = self.compressed_size();
if col_len < 0 || col_len < 0{
return Err(ParquetError::General(
"column start and length should not be negative".to_string(),
));
}
(col_start as u64, col_len as u64)
}
```

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by examining Column Chunk's byte_range() and the group-reader construction mentioned in the issue, then trace how corrupt metadata reaches Parquet reading. Add validation for invalid offsets or lengths and cover the corrupt-file case; the issue does not name a specific test or file to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.