lance-format / lance-format/lance
Support list string columns in MemWAL FTS
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
MemWAL in-memory FTS currently supports Utf8, LargeUtf8, and Utf8View columns. Persisted FTS indexes are gaining row-level document semantics for list string columns, so MemWAL should be evaluated and updated separately to keep fresh-tier FTS behavior consistent when list string rows are served from MemWAL.
Scope:
- Support
List<Utf8>,List<LargeUtf8>,LargeList<Utf8>, andLargeList<LargeUtf8>in the MemWAL FTS indexing path. - Use row-level document semantics: one list row is one document, and non-null elements are text fragments in that document.
- Keep null list rows, empty lists, and all-null element lists non-matching.
- Preserve the no-gap list position behavior if phrase queries are supported for this path.
- Add MemWAL FTS tests for list row-level matching and phrase matching across list elements.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the MemWAL FTS indexing path and its existing handling for Utf8, LargeUtf8, and Utf8View columns. Add tests for row-level matching and phrase matching across list elements, covering null rows, empty lists, and all-null element lists. Done means all four listed list-string types follow the stated document and position semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 67/100