[FEA] Add LZO decompression support to the ORC reader
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
The libcudf ORC reader recognizes the ORC `LZO` compression kind in the file footer, but rejects it because `is_supported_read_orc` does not include `compression_type::LZO` in its supported decompression codecs.
On the current `main` branch:
- `orc_decompressor` maps the ORC footer's `LZO` value to `compression_type::LZO`, then fails with `Unsupported compression type for ORC reader`: https://github.com/NVIDIA/cudf/blob/e44e3cf6a24cd075f022a3755a7546f0a203ded4/cpp/src/io/orc/orc.cpp#L397-L413
- `is_supported_read_orc` currently allows ZLIB, SNAPPY, ZSTD, and LZ4, but not LZO: https://github.com/NVIDIA/cudf/blob/e44e3cf6a24cd075f022a3755a7546f0a203ded4/cpp/src/io/functions.cpp#L326-L334
This is observable through cudf-spark. Spark 3.3 can write and read an LZO-compressed ORC dataset on CPU. With RAPIDS enabled, the read is planned as a GPU ORC scan and fails deterministically in libcudf.
The previously executed cudf-spark reproduction captured the following GPU/CPU result from the same Spark session:
```text
RAPIDS Enabled = YES
Plugins Loaded = YES
GPU Operators: YES
SUMMARY_CASE_ID = ORC_LZO
GPU_STATUS = EXCEPTION
GPU_EXCEPTION_CLASS = org.apache.spark.SparkException
GPU_ROOT_CAUSE_CLASS = ai.rapids.cudf.CudfException
GPU_ROOT_CAUSE_MESSAGE = CUDF failure at: ../../../thirdparty/cudf/cpp/src/io/orc/orc.cpp:413: Unsupported compression type for ORC reader
CPU_STATUS = SUCCESS
CPU_EXCEPTION_CLASS =
CPU_ROOT_CAUSE_CLASS =
CPU_ROOT_CAUSE_MESSAGE =
```
The executed GPU stack contains `OrcTableReader -> MakeOrcTableProducer -> MultiFileOrcPartitionReader`. The full reproduction command, Spark code, environment, and output are recorded in NVIDIA/cudf-spark#15551.
**Describe the solution you'd like**
Add LZO decompression support to the libcudf ORC reader and include ORC reader tests using LZO-compressed inputs. Once implemented, `is_supported_read_orc(compression_type::LZO)` should report support and LZO-compressed ORC data should produce the same results as a CPU ORC reader.
**Describe alternatives you've considered**
cudf-spark could inspect the ORC compression kind before selecting the GPU reader and fall back to Spark's CPU ORC reader for LZO files. That would avoid the runtime failure, but it would not provide GPU-accelerated reads and would leave LZO as a libcudf ORC compatibility gap.
**Additional context**
- Reproduced cudf-spark failure: https://github.com/NVIDIA/cudf-spark/issues/15551
- Broader cudf-spark compression-format test tracking: https://github.com/NVIDIA/cudf-spark/issues/143
- Reproduction rate reported in cudf-spark#15551: 100%
- Original Spark test: `write and read - file source orc - codec: lzo` in `FileSourceCodecSuite`
Contributor guide
Research direction
Start in cpp/src/io/orc/orc.cpp around orc_decompressor and in cpp/src/io/functions.cpp around is_supported_read_orc to trace the existing codec handling. Find the ORC reader test suite and add LZO-compressed inputs alongside the existing codec cases. Done means LZO is reported as supported, the reader succeeds, and results match the CPU ORC reader.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100