apache / apache/paimon-rust

Support FileIndex generation for append writes

Open
#818 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
197
Forks
92
Avg merge
1d 16h
Merged PRs (30d)
91

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar.

### Description

Rust can read Bitmap and Bloom Filter FileIndexes, but normal append writes do not generate them: `DataFileWriter` leaves `extra_files` empty and `embedded_index` unset.

This is a write-side follow-up to #34 and #780. Reuse the existing index writers and format to:

- Parse index column declarations, per-column options, and `file-index.in-manifest-threshold`, with explicit errors for invalid or unsupported configurations.
- Build indexes per physical data file, preserving row order and NULL positions across batches and resetting state on file rolling.
- Store the complete serialized index in `embedded_index` when its size is at most the threshold; otherwise write a `.index` sidecar and register it in `extra_files`. Match Java's default of 500 B.
- Handle both foreground and background close paths. Return metadata only after data and index completion succeeds, and clean up newly created sidecars on failure.

Tests should cover unconfigured writes, Bitmap/Bloom, multiple columns, NULLs, empty batches, rolling, threshold boundaries, and failure/abort cleanup. After commit, verify identical query results with indexes enabled and disabled, actual FileIndex pruning, and residual filtering for Bloom false positives.

Scope is limited to ordinary append writes and supported top-level columns. KV writes, COW/DataEvolution rewrites, dedicated Blob/Vector paths, nested indexes, additional index types, and public API expansion are excluded. Java-generated table fixtures remain a separate read-compatibility follow-up.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with DataFileWriter and the existing Bitmap and Bloom Filter index writers and format, using the write-side follow-ups in #34 and #780 for context. Add coverage for the listed write, rolling, threshold, failure, and cleanup cases; done means append writes produce correct embedded or sidecar indexes and query results, pruning, and Bloom residual filtering remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.