NVIDIA / NVIDIA/cudf

[FEA] Add LZO decompression support to the ORC reader

Open
#24,051 0 comments 0 reactions 0 assignees View on GitHub
? - Needs Triage cuIO feature request libcudf Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

The libcudf ORC reader recognizes the ORC `LZO` compression kind in the file footer, but rejects it because `is_supported_read_orc` does not include `compression_type::LZO` in its supported decompression codecs.

On the current `main` branch:

- `orc_decompressor` maps the ORC footer's `LZO` value to `compression_type::LZO`, then fails with `Unsupported compression type for ORC reader`: https://github.com/NVIDIA/cudf/blob/e44e3cf6a24cd075f022a3755a7546f0a203ded4/cpp/src/io/orc/orc.cpp#L397-L413
- `is_supported_read_orc` currently allows ZLIB, SNAPPY, ZSTD, and LZ4, but not LZO: https://github.com/NVIDIA/cudf/blob/e44e3cf6a24cd075f022a3755a7546f0a203ded4/cpp/src/io/functions.cpp#L326-L334

This is observable through cudf-spark. Spark 3.3 can write and read an LZO-compressed ORC dataset on CPU. With RAPIDS enabled, the read is planned as a GPU ORC scan and fails deterministically in libcudf.

The previously executed cudf-spark reproduction captured the following GPU/CPU result from the same Spark session:

```text
RAPIDS Enabled = YES
Plugins Loaded = YES
GPU Operators: YES

SUMMARY_CASE_ID = ORC_LZO
GPU_STATUS = EXCEPTION
GPU_EXCEPTION_CLASS = org.apache.spark.SparkException
GPU_ROOT_CAUSE_CLASS = ai.rapids.cudf.CudfException
GPU_ROOT_CAUSE_MESSAGE = CUDF failure at: ../../../thirdparty/cudf/cpp/src/io/orc/orc.cpp:413: Unsupported compression type for ORC reader
CPU_STATUS = SUCCESS
CPU_EXCEPTION_CLASS =
CPU_ROOT_CAUSE_CLASS =
CPU_ROOT_CAUSE_MESSAGE =
```

The executed GPU stack contains `OrcTableReader -> MakeOrcTableProducer -> MultiFileOrcPartitionReader`. The full reproduction command, Spark code, environment, and output are recorded in NVIDIA/cudf-spark#15551.

**Describe the solution you'd like**

Add LZO decompression support to the libcudf ORC reader and include ORC reader tests using LZO-compressed inputs. Once implemented, `is_supported_read_orc(compression_type::LZO)` should report support and LZO-compressed ORC data should produce the same results as a CPU ORC reader.

**Describe alternatives you've considered**

cudf-spark could inspect the ORC compression kind before selecting the GPU reader and fall back to Spark's CPU ORC reader for LZO files. That would avoid the runtime failure, but it would not provide GPU-accelerated reads and would leave LZO as a libcudf ORC compatibility gap.

**Additional context**

- Reproduced cudf-spark failure: https://github.com/NVIDIA/cudf-spark/issues/15551
- Broader cudf-spark compression-format test tracking: https://github.com/NVIDIA/cudf-spark/issues/143
- Reproduction rate reported in cudf-spark#15551: 100%
- Original Spark test: `write and read - file source orc - codec: lzo` in `FileSourceCodecSuite`

Contributor guide

Open the contributing guide

Research direction

Start in cpp/src/io/orc/orc.cpp around orc_decompressor and in cpp/src/io/functions.cpp around is_supported_read_orc to trace the existing codec handling. Find the ORC reader test suite and add LZO-compressed inputs alongside the existing codec cases. Done means LZO is reported as supported, the reader succeeds, and results match the CPU ORC reader.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.