typesense / typesense/typesense
`token_separators` in Query and Indexing
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 26.6k
- Forks
- 973
- Avg merge
- 18h 45m
- Merged PRs (30d)
- 4
Description
Here is my setup:
{
"name": "my_collection",
"fields": [
{
"name": "name",
"type": "string",
"token_separators": ["-"]
}
]
}
I'm creating a document with the name "foo-bar-baz":
client.collections['my_collection'].documents.create(
{
"id": "001",
"name": "foo-bar-baz"
}
)
The Issue
When I search for the document using the hyphenated query "foo-bar-baz", it returns no results. However, when I manually change the query to "foo bar baz" (with spaces), the record is successfully returned.
// This search returns no result
client.collections['my_collection'].documents.search(
{
"q": "foo-bar-baz",
"query_by": "name"
}
)
// This search returns the record
client.collections['my_collection'].documents.search(
{
"q": "foo bar baz",
"query_by": "name"
}
)
My Question
My understanding was that token_separators would apply to both the indexed data and the search query, causing both "foo-bar-baz" and "foo bar baz" to be tokenized the same way and return a result.
Am I missing a parameter or setting that needs to be applied to the search query? Or do I need to manually tokenize the search query on the client-side before sending it to Typesense?
Any guidance on the expected behavior would be greatly appreciated. Thank you!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported behavior using the collection field's token_separators setting, then compare indexing of "foo-bar-baz" with query parsing for the same text and for "foo bar baz". The issue names no source files or tests; done means establishing the expected query behavior and adding a regression test or documenting the required setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100