Support ORC batch size
Open
ORC
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 21
Description
**Is your feature request related to a problem? Please describe.**
Now `spark.auron.batchSize` defaults to 10000. ORC may report an error when reading such large data.
```
`Execution error: offset overflow`
```
orc-rust src/array_decoder/string.rs:111
```rust
let offsets =
OffsetBuffer::::from_lengths(lengths.into_iter().map(|l| l as usize));
```
**Describe the solution you'd like**
Implement a Spark-like configuration `spark.sql.orc.columnarReaderBatchSize`, which can individually control the batch size of ORC.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.