lance-format / lance-format/lance
Tokenizer plugins
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
It won't be sustainable to statically compile tokenizers into Lance. It also would be annoying to require them all to have feature flags in Lance.
We currently use Tantivy for tokenizer, so it might make sense for us to re-use their API. There is some discussion on their side for using a plugin API. It might make sense for us to help with that and re-use that same mechanism. That way tokenizer providers don't have to do anything extra to support Lance if they are supporting tantivy.
Tantivy has a stable API for tokenizeres here: https://github.com/quickwit-oss/tantivy/blob/main/tokenizer-api/src/lib.rs
(It's a Rust API though. We might want to create a stable C API, which will be more cross compatible between compilers.)
https://github.com/quickwit-oss/tantivy-py/issues/358
https://github.com/quickwit-oss/tantivy-py/pull/200
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading Tantivy's tokenizer-api/src/lib.rs and the linked tantivy-py discussion and pull request. Determine whether Lance should reuse Tantivy's plugin mechanism or define a stable C API, and document the compatibility boundary and provider requirements before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100