lance-format / lance-format/lance
Generated Columns
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
We are using lance to store text corpus, but before training, some lightweight normalizations should be applied to the text, for example, removing sensitive words. currently, we have to store both normalized text and unnormalized text. the storage is doubled.
maybe lance can implement generated column which is similar to https://www.sqlite.org/gencol.html.
the benifits are:
- save storage size
- better IO performance
- no need to update normalized text any more.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the generated-column request and the linked SQLite generated-column reference. Determine how the proposed behavior would apply to Lance columns and text normalization, with success measured by avoiding duplicate normalized and unnormalized text storage while keeping normalized text available before training.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100