CentreForDigitalHumanities / CentreForDigitalHumanities/idioms

Allow matching substrings in free text search?

Open
#28 1 comment 0 reactions 0 assignees View on GitHub
enhancement postponed
Dominant language
HTML
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

The FTS5 trigram tokenizer allows matching substrings in general (rather than only complete tokens or prefix tokens), but it does not match substrings shorter than 3 unicode characters. Supporting short tokens is required (e.g. general structure contains mainly short forms like `V DO`), so if substring matching is desired a different solution than SQLite FTS is needed.

Prefix tokens are supported by the (default) `unicode61` tokenizer, which is used now.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the issue's notes on SQLite FTS5, the trigram tokenizer, and the default unicode61 tokenizer. Research an alternative search approach that supports substring matches shorter than three Unicode characters, and verify that examples such as `V DO` are matched when the work is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite
Domain
database, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.