apache / apache/arrow

[C++][Parquet] Decoding: allow Boolean RecordReader get raw LSB bitmap

Open
#39,227 34 comments 0 reactions 1 assignee Claimed by @mapleFU View on GitHub
Component: C++ Component: Parquet good-first-issue Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Plain Boolean Decoding is far more slower than Rle Boolean Decoding. This is because:

1. `PlainBooleanDecoder` uses `BitReader::GetBatch` to decoding Bool
2. `BitReader::GetBatch` is optimized for 32bits and 64bits input with unpack32/unpack64
3. However, when input is `bool`, the code will fallback to the logic:

```c++
if (sizeof(T) == 4) {
int num_unpacked =
internal::unpack32(reinterpret_cast(buffer + byte_offset),
reinterpret_cast(v + i), batch_size - i, num_bits);
i += num_unpacked;
byte_offset += num_unpacked * num_bits / 8;
} else if (sizeof(T) == 8 && num_bits > 32) {
// Use unpack64 only if num_bits is larger than 32
// TODO (ARROW-13677): improve the performance of internal::unpack64
// and remove the restriction of num_bits
int num_unpacked =
internal::unpack64(buffer + byte_offset, reinterpret_cast(v + i),
batch_size - i, num_bits);
i += num_unpacked;
byte_offset += num_unpacked * num_bits / 8;
} else {
// TODO: revisit this limit if necessary
DCHECK_LE(num_bits, 32);
const int buffer_size = 1024;
uint32_t unpack_buffer[buffer_size];
while (i < batch_size) {
int unpack_size = std::min(buffer_size, batch_size - i);
int num_unpacked =
internal::unpack32(reinterpret_cast(buffer + byte_offset),
unpack_buffer, unpack_size, num_bits);
if (num_unpacked == 0) {
break;
}
for (int k = 0; k < num_unpacked; ++k) {
#ifdef _MSC_VER
#pragma warning(push)
#pragma warning(disable : 4800)
#endif
v[i + k] = static_cast(unpack_buffer[k]);
#ifdef _MSC_VER
#pragma warning(pop)
#endif
}
i += num_unpacked;
byte_offset += num_unpacked * num_bits / 8;
}
}
```

Maybe we can specialize the case with `sizeof(T) == 1` to optimize this?

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.