heatherleaf / heatherleaf/korpsearch
n-gram character index
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Currently the indexes are on token level:
- singleton index:
[lemma]or[pos] - pairwise index:
[pos][lemma]or[pos][][pos] - etc
Inspired by plocate, we could create character indexes instead (3-grams or 4-grams or ...). Where the characters are for lemmas, pos, etc, or combinations.
Instead of the entries [lemma="car"], [lemma="card"], [lemma="cart"], and [lemma="ward"] we would get:
[lemma="*car*"],[lemma="*ard*"],[lemma="*art*"], and[lemma="*war*"]
This would only be interesting for some features (lemma, word, etc)
For combined features: [pos="ADJ"][lemma="*car*"], [pos="DET"][lemma="*art*"], [lemma="*war*"][pos="NN"], etc.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing token-level index implementation and how queries for lemma, word, POS, and combined features are handled. Define which features and n-gram sizes are supported, then verify that the example character-index queries produce correct search results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100