Check valid byte_range in parquet Column Chunk reading?
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
See: https://github.com/apache/parquet-testing/pull/58#issuecomment-2290985490
When reading a corrupt file, currently, arrow-rs would have:
```rust
/// Returns the offset and length in bytes of the column chunk within the file
pub fn byte_range(&self) -> (u64, u64) {
let col_start = match self.dictionary_page_offset() {
Some(dictionary_page_offset) => dictionary_page_offset,
None => self.data_page_offset(),
};
let col_len = self.compressed_size();
assert!(
col_start >= 0 && col_len >= 0,
"column start and length should not be negative"
);
(col_start as u64, col_len as u64)
}
```
Would we better check the range here?
**Describe the solution you'd like**
Checking the range when building the group reader or in "byte_range()"
**Describe alternatives you've considered**
```rust
/// Returns the offset and length in bytes of the column chunk within the file
pub fn byte_range(&self) -> Result<(u64, u64)> {
let col_start = match self.dictionary_page_offset() {
Some(dictionary_page_offset) => dictionary_page_offset,
None => self.data_page_offset(),
};
let col_len = self.compressed_size();
if col_len < 0 || col_len < 0{
return Err(ParquetError::General(
"column start and length should not be negative".to_string(),
));
}
(col_start as u64, col_len as u64)
}
```
**Additional context**
Contributor guide
Research direction
Start by examining Column Chunk's byte_range() and the group-reader construction mentioned in the issue, then trace how corrupt metadata reaches Parquet reading. Add validation for invalid offsets or lengths and cover the corrupt-file case; the issue does not name a specific test or file to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100