typesense / typesense/typesense
The `filter_by` is not working as expected for hybrid search
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 26.6k
- Forks
- 973
- Avg merge
- 18h 45m
- Merged PRs (30d)
- 4
Description
Bug Description
I expect the query to have 0 hit, nothing matched, but Typesense gives me result.
Reproduction Steps
#!/bin/bash
# sematic search with auto embedding
# filter by field
QUERY="nonexistent"
CHUNK=300
curl -s -X POST "http://172.16.8.107:8108/multi_search" \
-H "Content-Type: application/json" \
-H "X-Typesense-Api-Key: typesense" \
-d '{
"searches": [
{
"collection": "stock_reports_cn",
"q": "'${QUERY}'",
"query_by": "title,content,name,code,category,institution,embedding",
"per_page": 2,
"filter_by": "name:='宁德时代'",
"highlight_fields": "title,content,institution",
"highlight_affix_num_tokens": '${CHUNK}',
"sort_by": "_text_match:desc, date:desc",
"exclude_fields": "path,content,embedding"
},
{
"collection": "stock_notices_cn",
"q": "'${QUERY}'",
"query_by": "title,content,name,code,category,embedding",
"vector_query": "embedding:([], k:100, distance_threshold:0.2, alpha: 0.7)",
"per_page": 2,
"filter_by": "name:='宁德时代'",
"highlight_fields": "title,content",
"highlight_affix_num_tokens": '${CHUNK}',
"sort_by": "_text_match:desc, date:desc",
"exclude_fields": "path,content,embedding"
}
]
}'
Expected vs Actual
Expected behavior
nothing matched.
Actual behavior
Typesense gives me result.
{
"results": [
{
"facet_counts": [],
"found": 100,
"hits": [
{
"document": {
"category": "stock",
"code": "300750",
"created": 1773733034,
"date": 1742256000,
"id": "AP202503181644493316",
"institution": "交银国际证券有限公司",
"name": "宁德时代",
"title": "全球扩产提速,盈利能力保持韧性;维持买入"
},
"highlight": {},
"highlights": [],
"hybrid_search_info": {
"rank_fusion_score": 0.30000001192092896
},
"text_match": 0,
"text_match_info": {
"best_field_score": "0",
"best_field_weight": 0,
"fields_matched": 0,
"num_tokens_dropped": 1,
"score": "0",
"tokens_matched": 0,
"typo_prefix_score": 255
},
"vector_distance": 0.2201164960861206
},
{
"document": {
"category": "stock",
"code": "300750",
"created": 1773739651,
"date": 1729555200,
"id": "AP202410221640410402",
"institution": "中银国际证券股份有限公司",
"name": "宁德时代",
"title": "盈利能力提升,海外进展顺利"
},
"highlight": {},
"highlights": [],
"hybrid_search_info": {
"rank_fusion_score": 0.15000000596046448
},
"text_match": 0,
"text_match_info": {
"best_field_score": "0",
"best_field_weight": 0,
"fields_matched": 0,
"num_tokens_dropped": 1,
"score": "0",
"tokens_matched": 0,
"typo_prefix_score": 255
},
"vector_distance": 0.2214726209640503
}
],
"out_of": 119314,
"page": 1,
"request_params": {
"collection_name": "stock_reports_cn",
"first_q": "nonexistent",
"per_page": 2,
"q": "nonexistent"
},
"search_cutoff": false,
"search_time_ms": 164
},
{
"facet_counts": [],
"found": 0,
"hits": [],
"out_of": 55489,
"page": 1,
"request_params": {
"collection_name": "stock_notices_cn",
"first_q": "nonexistent",
"per_page": 2,
"q": "nonexistent"
},
"search_cutoff": false,
"search_time_ms": 78
}
]
}
Environment
- Typesense version: 30.1
- Operating system: Ubuntu 22.04.4 LTS
- Client library & version: rest api
Schema / Configuration
# create research collection with auto-embedding fields
curl -X POST http://172.16.8.107:8108/collections \
-H "X-TYPESENSE-API-KEY: typesense" \
-H "Content-Type: application/json" \
-d '{
"name": "stock_reports_cn",
"fields": [
{"name": "id", "type": "string"},
{"name": "date", "type": "int64", "sort": true},
{"name": "code", "type": "string"},
{"name": "name", "type": "string", "locale": "zh"},
{"name": "category", "type": "string", "locale": "zh"},
{"name": "institution", "type": "string", "locale": "zh"},
{"name": "title", "type": "string", "locale": "zh"},
{"name": "content", "type": "string", "locale": "zh"},
{"name": "created", "type": "int64", "sort": true},
{"name": "embedding", "type": "float[]", "embed": {
"from": ["title", "content"],
"model_config": {
"model_name": "ts/gte-large"
}
}
}
],
"default_sorting_field": "date"
}'
# create notice collection with auto-embedding fields
curl -X POST http://172.16.8.107:8108/collections \
-H "X-TYPESENSE-API-KEY: typesense" \
-H "Content-Type: application/json" \
-d '{
"name": "stock_notice_cn",
"fields": [
{"name": "id", "type": "string"},
{"name": "date", "type": "int64", "sort": true},
{"name": "code", "type": "string"},
{"name": "name", "type": "string", "locale": "zh"},
{"name": "category", "type": "string", "locale": "zh"},
{"name": "title", "type": "string", "locale": "zh"},
{"name": "content", "type": "string", "locale": "zh"},
{"name": "created", "type": "int64", "sort": true},
{"name": "embedding", "type": "float[]", "embed": {
"from": ["title", "content"],
"model_config": {
"model_name": "ts/gte-large"
}
}
}
],
"default_sorting_field": "date"
}'
Additional Context
The highlights worked as expected, nonexistent keyword should match nothing, but the filter dose not work as expected.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported curl request against the REST multi_search endpoint, comparing the filter_by behavior in the stock_reports_cn hybrid search with the stock_notices_cn vector search. Trace how filter_by is applied alongside vector_query and hybrid ranking; done means the name filter excludes all non-matching results and a regression test covers this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend-api-design, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100