typesense / typesense/typesense

`token_separators` in Query and Indexing

Open
#2,556 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
C++
Stars
26.6k
Forks
973
Avg merge
18h 45m
Merged PRs (30d)
4

Description

Here is my setup:

{
  "name": "my_collection",
  "fields": [
    {
      "name": "name",
      "type": "string",
      "token_separators": ["-"]
    }
  ]
}

I'm creating a document with the name "foo-bar-baz":

client.collections['my_collection'].documents.create(
  {
    "id": "001",
    "name": "foo-bar-baz"
  }
)

The Issue

When I search for the document using the hyphenated query "foo-bar-baz", it returns no results. However, when I manually change the query to "foo bar baz" (with spaces), the record is successfully returned.

// This search returns no result
client.collections['my_collection'].documents.search(
  {
    "q": "foo-bar-baz",
    "query_by": "name"
  }
)

// This search returns the record
client.collections['my_collection'].documents.search(
  {
    "q": "foo bar baz",
    "query_by": "name"
  }
)

My Question

My understanding was that token_separators would apply to both the indexed data and the search query, causing both "foo-bar-baz" and "foo bar baz" to be tokenized the same way and return a result.

Am I missing a parameter or setting that needs to be applied to the search query? Or do I need to manually tokenize the search query on the client-side before sending it to Typesense?

Any guidance on the expected behavior would be greatly appreciated. Thank you!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported behavior using the collection field's token_separators setting, then compare indexing of "foo-bar-baz" with query parsing for the same text and for "foo bar baz". The issue names no source files or tests; done means establishing the expected query behavior and adding a regression test or documenting the required setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.