Support FileIndex generation for append writes
- Dominant language
- Rust
- Stars
- 197
- Forks
- 92
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 91
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar.
### Description
Rust can read Bitmap and Bloom Filter FileIndexes, but normal append writes do not generate them: `DataFileWriter` leaves `extra_files` empty and `embedded_index` unset.
This is a write-side follow-up to #34 and #780. Reuse the existing index writers and format to:
- Parse index column declarations, per-column options, and `file-index.in-manifest-threshold`, with explicit errors for invalid or unsupported configurations.
- Build indexes per physical data file, preserving row order and NULL positions across batches and resetting state on file rolling.
- Store the complete serialized index in `embedded_index` when its size is at most the threshold; otherwise write a `.index` sidecar and register it in `extra_files`. Match Java's default of 500 B.
- Handle both foreground and background close paths. Return metadata only after data and index completion succeeds, and clean up newly created sidecars on failure.
Tests should cover unconfigured writes, Bitmap/Bloom, multiple columns, NULLs, empty batches, rolling, threshold boundaries, and failure/abort cleanup. After commit, verify identical query results with indexes enabled and disabled, actual FileIndex pruning, and residual filtering for Bloom false positives.
Scope is limited to ordinary append writes and supported top-level columns. KV writes, COW/DataEvolution rewrites, dedicated Blob/Vector paths, nested indexes, additional index types, and public API expansion are excluded. Java-generated table fixtures remain a separate read-compatibility follow-up.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
Research direction
Start with DataFileWriter and the existing Bitmap and Bloom Filter index writers and format, using the write-side follow-ups in #34 and #780 for context. Add coverage for the listed write, rolling, threshold, failure, and cleanup cases; done means append writes produce correct embedded or sidecar indexes and query results, pruning, and Bloom residual filtering remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100