apache / apache/hudi-rs

cpp: read_file_slice{,_from_paths} ignore ReadOptions (no projection/filter pushdown)

Open
#637 0 comments 0 reactions 1 assignee Claimed by @codope View on GitHub
Dominant language
Rust
Stars
279
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
40

Description

## Summary

The C++ FFI's slice-read entry points in `cpp/src/lib.rs` — `read_file_slice` and `read_file_slice_from_paths` — construct `ReadOptions::new()` internally and pass that to the inner reader. As a result, any read options the caller might want (column projection, filters, read-optimized mode, batch size) never reach the read.

The Rust core already supports these: `FileGroupReader::read_file_slice_from_paths` honors `options.projection` and filters and applies them to the merged result. Only the C++ shim drops them.

## Impact for engine integrations

Engines embedding the C++ library (e.g. Presto native / Velox — see apache/hudi#18308) get no column pruning or predicate pushdown through the FFI. The merged batch materializes all columns for the slice even when the query needs a few, which is both slower and heavier on memory (a MOR slice with logs is merged in memory and returned as a single batch).

## Proposal

Widen the C++ slice-read entry points to accept read options and forward them to the inner reader. A simple, `cxx`-friendly shape is a `Vec` of `key=value` options (consistent with `new_file_group_reader_with_options`), with a reserved `projection=col1,col2` key mapped to `ReadOptions::with_projection`; other keys map to `with_hudi_option`. A dedicated projection parameter would also work.

This enables column projection immediately and leaves a clear path to filter pushdown later, without further FFI signature changes.

## Notes

Pairs with the panic-safety hardening tracked separately. Both are small, isolated changes to `cpp/src/lib.rs` and are needed for a usable query-engine integration.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.