CentreForDigitalHumanities / CentreForDigitalHumanities/idioms
Allow matching substrings in free text search?
- Dominant language
- HTML
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
The FTS5 trigram tokenizer allows matching substrings in general (rather than only complete tokens or prefix tokens), but it does not match substrings shorter than 3 unicode characters. Supporting short tokens is required (e.g. general structure contains mainly short forms like `V DO`), so if substring matching is desired a different solution than SQLite FTS is needed.
Prefix tokens are supported by the (default) `unicode61` tokenizer, which is used now.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the issue's notes on SQLite FTS5, the trigram tokenizer, and the default unicode61 tokenizer. Research an alternative search approach that supports substring matches shorter than three Unicode characters, and verify that examples such as `V DO` are matched when the work is complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sqlite
- Domain
- database, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100