[FEA] Explore adding Vortex file format support to cuDF
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
## Context
[Vortex](https://github.com/vortex-data/vortex) is an extensible, state-of-the-art framework for columnar compression and the fastest FOSS columnar file format. Formerly at [@spiraldb](https://github.com/spiraldb), Vortex is now an Incubation Stage project at [LFAI & Data](https://lfaidata.foundation/), part of the Linux Foundation.
Vortex provides GPU-native decoding and exports GPU-resident arrays through the [Arrow C Device Interface](https://arrow.apache.org/docs/format/CDeviceDataInterface.html). Existing [interoperability tests](https://github.com/vortex-data/vortex/tree/develop/vortex-test/e2e-cuda) already exercise this path with cuDF.
This issue is intended to work with the cuDF maintainers on a rough integration path. The APIs and build integration below are starting points for discussion rather than a finalized design.
## Possible Python API
```python
df = cudf.read_vortex(
"dataset.vortex",
columns=["id", "timestamp", "value"],
filters=[("timestamp", ">=", cutoff)],
)
```
The reader would return a `cudf.DataFrame` directly, support projection and predicate pushdown, and avoid staging decoded data through host-backed Arrow, pandas, or NumPy.
## Proposed integration direction
### Vortex C++ layer
Vortex is primarily implemented in Rust and does not yet expose the full C++ reader API needed by libcudf. We expect to develop this API under [`lang/cpp`](https://github.com/vortex-data/vortex/tree/develop/lang/cpp), likely backed by a stable C ABI.
The existing CMake configuration in `lang/cpp` is currently bare-bones and would be expanded to:
- Build and link the required Vortex Rust and CUDA components.
- Expose installable CMake targets consumable by cuDF/RAPIDS.
- Support Linux `x86_64` and `aarch64`.
- Provide file reading, projection, predicate pushdown, and CUDA stream handling.
- Export `ArrowDeviceArray` and `ArrowSchema` with explicit ownership semantics.
The exact packaging and dependency mechanism should be agreed on with the cuDF maintainers.
### libcudf
A libcudf integration could add:
- `cudf::io::vortex_reader_options`
- `cudf::io::read_vortex`
The reader would pass projection and predicates to Vortex, then import the resulting Arrow device arrays directly into libcudf. It should preserve schema metadata, respect cuDF stream and RMM semantics, and avoid host materialization.
### Python cuDF
The libcudf reader could be exposed through pylibcudf/Cython as `cudf.read_vortex`, following existing cuDF I/O conventions for `columns=`, `filters=`, and applicable storage options.
## Questions for cuDF maintainers
- Is the Arrow C Device Interface an appropriate boundary between Vortex and libcudf?
- What libcudf reader API shape would fit existing I/O conventions?
- How should Vortex be discovered and built through RAPIDS/CMake?
- What are the expected CUDA stream, RMM, ownership, and fallback semantics?
- What would be a reasonable first integration milestone?
## Existing GPU support
Vortex has CUDA execution for ALP, BitPacked/FastLanes, DateTimeParts, DecimalByteParts, Delta, Dictionary, Frame of Reference, FSST, OnPair, RunEnd, Sequence, ZigZag, and Zstandard via nvCOMP. It also supports constant, list, masked, and shared arrays, plus filter and slice operations.
## Performance
CUDA kernel microbenchmarks on an **NVIDIA GH200**, using Vortex commit [`1fe8dda`](https://github.com/vortex-data/vortex/commit/1fe8dda030764bb3ac6faed21182fb968b516529):
- **BitPacked:** **1,298–2,711 GiB/s**.
- **RunEnd:** up to **1,931 GiB/s**.
- **Dictionary:** **1,442–2,265 GiB/s**.
- **ALP:** **1,372 GiB/s** for `f32` and **1,629 GiB/s** for `f64`.
- **FSST:** **248–281 GiB/s** across decoding and Arrow C Device export.
These are kernel benchmarks, not end-to-end file-read results.
Contributor guide
Research direction
Start by reading cuDF’s existing I/O conventions and the proposed Vortex lang/cpp CMake setup, then inspect the referenced Vortex CUDA interoperability tests. Review how a libcudf reader could connect to pylibcudf/Cython and the Arrow C Device Interface. Done means agreeing on a first integration milestone, API shape, build path, and stream, ownership, and fallback semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp, python, rust
- Domain
- backend-api-design, build-system, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100