apache / apache/paimon-rust

[Feature] Support primary-key table full-text and hybrid search (read side)

Open
#567 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
197
Forks
92
Avg merge
1d 16h
Merged PRs (30d)
91

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar.

### Motivation

Paimon (Java) supports full-text search and hybrid (vector + full-text) search on primary-key tables, backed by the shared native `paimon-ftindex-core` engine. The Rust read side already supports primary-key **vector** search end-to-end; the natural next step is to read the primary-key **full-text** and **hybrid** results Java writes, so that Rust / Doris / DataFusion consumers get the same primary-key search capabilities.

### Solution

Mirror the Java PK read path, reusing the already-merged PK-vector machinery and depending on `paimon-ftindex-core` (the same core the Java/Python bindings use), gated behind the `fulltext` feature:

- Read `full-text` index archives via `paimon-ftindex-core` (draft PR #563 lands the reader foundation).
- PK full-text data layer (config options + the shared `index/pk` source-metadata + bucket-state reconciliation) and read path (`PrimaryKeyFullTextScan` / `PrimaryKeyFullTextBucketSearch` / `PrimaryKeyFullTextRead`, FAST mode), materializing rows by physical position — mirroring Java.
- Hybrid search on primary-key tables: fuse vector + full-text routes in physical-position space via a `PrimaryKeySearchRanker` (RRF / weighted-score / weighted-MRR), mirroring Java `HybridSearchBuilderImpl`.

This tracks the same effort as #563 and its follow-ups. I have the read path (foundation → data layer → PK full-text read → PK hybrid) implemented locally and passing the full test suite; it is gated on the `paimon-ftindex-core` crates.io release before the PRs can be marked ready (a git-only dependency blocks `cargo package`).

### Anything else?

Related: PK-vector read tracking (#514) and draft PR #563. Depends on `apache/paimon-full-text` (`paimon-ftindex-core`) being published to crates.io.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with draft PR #563 and the existing primary-key vector read machinery, then trace the shared index/pk source-metadata and bucket-state handling. Review the planned PrimaryKeyFullTextScan, PrimaryKeyFullTextBucketSearch, PrimaryKeyFullTextRead, and PrimaryKeySearchRanker paths. Done means full-text and hybrid primary-key results work behind the fulltext feature and the full test suite passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.