lance-format / lance-format/lance-c

Compound FtsQuery C ABI surface — design preference (JSON vs typed handles)?

Open
#35 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
5
Forks
14
Avg merge
3d 5h
Merged PRs (30d)
18

Description

Hi! We're integrating lance-c as the items / FTS storage backend for an internal C++ service and want to understand early on the surface for compound FTS queries.

The Phase 2 design doc already names this as future work (docs/superpowers/specs/2026-04-23-phase2-vector-search-indexing-design.md:22):

Compound boolean FTS queries (Boost / Boolean / Phrase composition). MVP
exposes match + fuzzy; the composer can be added later without breaking
changes.

We'd like to know the rough timeline for the implementation and happy to help land this. What's your preferred shape for the C ABI?

What we need

Concretely, we need to express each FtsQuery variant from lance_index::scalar::inverted::query over the C ABI:

Need (downstream consumer) Maps to
AND across query terms MatchQuery::with_operator(And)
Type-ahead (last token as prefix) + expansion cap prefix expansion → BooleanQuery(Should) of MatchQuery::with_max_expansions(...)
Per-attribute boost (title^1.2 body) MultiMatchQuery or BooleanQuery(Should) of MatchQuery::with_boost
Sub-queries with per-clause match_all_terms / last_as_prefix / boost BooleanQuery(Should) of distinct MatchQuery
Phrase queries with slop PhraseQuery::with_slop
Negative-boost re-rank BoostQuery

The current C function:

int32_t lance_scanner_full_text_search(
    LanceScanner* scanner,
    const char* query,
    const char* const* columns,
    uint32_t max_fuzzy_distance);

covers MatchQuery::new + MatchQuery::with_fuzziness(Some(d)) only.

Two surface shapes we're considering

Option A — JSON entrypoint. Single new function:

/// Set a serialized FtsQuery (lance_index::scalar::inverted::query::FtsQuery)
/// as JSON. The Rust types already derive Serialize/Deserialize.
int32_t lance_scanner_fts_query_json(
    LanceScanner* scanner,
    const char* fts_query_json,
    size_t json_len);

Pros: one C symbol; forward-compatible with new variants for free;
implementation is serde_json::from_slice + set_fts_query.
Cons: callers serialize JSON; errors surface late (deserialization), not at
call sites; less idiomatic vs the typed nearest/nprobes/refine_factor
chain pattern.

Option B — Typed handles. Opaque LanceFtsQuery* plus builders:

LanceFtsQuery* lance_fts_match_new(const char* terms);
int32_t lance_fts_match_set_column(LanceFtsQuery*, const char*);
int32_t lance_fts_match_set_operator(LanceFtsQuery*, LanceFtsOperator);
int32_t lance_fts_match_set_fuzziness(LanceFtsQuery*, int32_t);
int32_t lance_fts_match_set_max_expansions(LanceFtsQuery*, uint32_t);
int32_t lance_fts_match_set_boost(LanceFtsQuery*, float);
int32_t lance_fts_match_set_prefix_length(LanceFtsQuery*, uint32_t);

LanceFtsQuery* lance_fts_phrase_new(const char* terms);
int32_t lance_fts_phrase_set_slop(LanceFtsQuery*, uint32_t);

LanceFtsQuery* lance_fts_boost_new(LanceFtsQuery* positive, LanceFtsQuery* negative, float boost);
LanceFtsQuery* lance_fts_multi_match_new(const LanceFtsQuery* const* matches, size_t n);
LanceFtsQuery* lance_fts_boolean_new(void);
int32_t lance_fts_boolean_add(LanceFtsQuery*, LanceFtsOccur, LanceFtsQuery*);

void lance_fts_query_close(LanceFtsQuery*);

int32_t lance_scanner_fts_query(LanceScanner*, LanceFtsQuery*);

Pros: matches the existing Scanner::nearest + nprobes/refine_factor/ef pattern; errors surface at construction time; idiomatic in C++ via RAII wrappers in lance.hpp.
Cons: ~12 new C functions vs 1; new variants in the future need new
symbols (still ABI-compatible — just additive).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/superpowers/specs/2026-04-23-phase2-vector-search-indexing-design.md at the compound FTS query entry and inspect the existing lance_scanner_full_text_search API. Compare the listed FtsQuery variants with the current scanner builder pattern and the proposed JSON and typed-handle surfaces. Done means reaching and documenting a decided C ABI shape and implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, cpp, rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.