apache / apache/arrow

Parquet REL decoding w/IAA(Intel® In-Memory Analytics Accelerator)

Open
#14,684 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

We want to add new codes to enable a new RLE Decoding path by using Intel® In-Memory Analytics Accelerator (Intel® IAA). The new solution provides in general higher performance than the current solution, and also consumes less CPU.

Intel® In-Memory Analytics Accelerator (Intel® IAA) is a hardware accelerator available in the upcoming generation of Intel® Xeon® Scalable processors ("Sapphire Rapids"). Its goal is to speed up common operations in analytics like data (de)compression and filtering.

We are concerned about 3 issues:
1. Since IAA is a new accelerator in next-generation processors which will be published soon, the current machine running CI cannot support this feature. I don’t know if it is acceptable that we run the CI offline, maybe on Intel’s lab environment.
2. [Intel® Query Processing Library (QPL)](https://github.com/intel/qpl) is a library to access the hardware accelerator IAA, is it acceptable to added the dependency in the tool chain?
3. Do you think the work is valuable?

Thank you!

Contributor guide

Open the contributing guide

Research direction

The issue does not name implementation files, tests, or entry points. First clarify whether Intel IAA support and the Intel QPL dependency are acceptable, then identify the existing Parquet RLE decoding path and a CI or hardware test environment. Done means an agreed implementation scope and a way to validate performance and correctness.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.