typesense / typesense/typesense
Feature Request: General support for logographic languages (without specifying a field)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 26.6k
- Forks
- 973
- Avg merge
- 18h 45m
- Merged PRs (30d)
- 4
Description
Description
To use tokenizers for logographic languages we currently need to explicitly specify a field. This is a problem if you have multiple languages in a collection or you don't know the language of a document/field.
It would be great if the tokenizer can detect all supported languages that use a special tokenizer, use the appropriate tokenizer for that field/document and the fallback tokenizer for all others. Field-level detection is preferred.
Metadata
Typesense Version: v0.24.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by tracing tokenizer selection for fields and documents, then determine how supported logographic languages can be detected and assigned their appropriate tokenizer while other content uses the fallback; done means mixed-language collections work without an explicit field-level tokenizer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100