apache / apache/auron

Support ORC batch size

Open
#1,743 3 comments 0 reactions 1 assignee Claimed by @Sigma-Ma View on GitHub
ORC
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

**Is your feature request related to a problem? Please describe.**

Now `spark.auron.batchSize` defaults to 10000. ORC may report an error when reading such large data.

```
`Execution error: offset overflow`
```

orc-rust src/array_decoder/string.rs:111
```rust
let offsets =
OffsetBuffer::::from_lengths(lengths.into_iter().map(|l| l as usize));
```

**Describe the solution you'd like**

Implement a Spark-like configuration `spark.sql.orc.columnarReaderBatchSize`, which can individually control the batch size of ORC.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.