[FEA] Improve ORC reader performance for decimal types
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Decode of decimal files is an order of magnitude slower than decode of integral types.
The reason is the use of a single thread to find the sizes of the next batch of elements, which are then decoded using the whole block. To improve kernel performance, it needs to use multiple threads to find varint boundaries:
Pass 1: every thread runs `is_boundary_byte` (highest bit == 0) to find if it's at the last byte of a varint element.
```
A = 0 0 0 0 1 0 0 1 0 0 0 1
```
Pass 2: Scan A to produce B. Also gets the number of elements (3 in this case).
```
B = 0 0 0 0 1 1 1 2 2 2 2 3
^ ^ ^
t0 t4 t7
```
Pass 3: Threads that are on a boundary decode the element that starts at their index and store it at col[t].
t=0 writes to [0]
t=4 writes to [1]
t=7 writes to [2]
Alternatively, step 3 can store the offsets of each element so they can be decoded in parallel.
Contributor guide
Assessment
This issue has not been assessed yet.