Support Elasticsearch field collapse (deduplication by documentId) in HybridSearch
- Dominant language
- C#
- Stars
- 21
- Forks
- 4
- Avg merge
- 1h 18m
- Merged PRs (30d)
- 1
Description
**Description**
We are using Microsoft.Extensions.VectorData with Elasticsearch as the vector store and calling:
`IKeywordHybridSearchable.HybridSearchAsync(...)`
Our data is chunked (multiple records per logical document). Each chunk shares the same documentId.
During hybrid search, multiple chunks from the same document are returned, which results in duplicate documents at the application level. Elasticsearch natively supports solving this using the collapse parameter:
```
"collapse": {
"field": "documentId",
"inner_hits": {
"size": 1,
"sort": [{ "_score": "desc" }]
}
}
```
However, HybridSearchOptions currently exposes only:
1. Filter
2. VectorProperty
3. AdditionalProperty
4. Skip
5. IncludeVectors
There is no way to specify collapse or grouping options, and collapse cannot be expressed via Filter.
**Problem Statement**
When using chunked indexing:
Hybrid search returns multiple chunks from the same document
Consumers must manually deduplicate results
This is inefficient and differs from Elasticsearch-native behavior
We need a way to request one result per logical document directly at the Elasticsearch query
**Proposed Enhancement**
Add optional collapse support to HybridSearchOptions, for example:
```
public class HybridSearchOptions
{
...
public string? CollapseField { get; set; } // e.g. "documentId"
public int? CollapseInnerHitsSize { get; set; } = 1;
}
```
Then, in the Elasticsearch connector, map this to:
```
"collapse": {
"field": "",
"inner_hits": {
"size":
}
}
```
**Benefits**
Prevents duplicate documents when using chunked storage
Aligns hybrid search behavior with native Elasticsearch capabilities
Improves relevance, performance, and RAG evidence quality
Removes need for application-side deduplication
**Could you please advise where in the Elasticsearch connector/library we should add support for the collapse parameter (or an equivalent option) for HybridSearchAsync, or if there is an existing extension point or recommended approach to implement Elasticsearch field collapse (e.g., by documentId) without forking the entire library?**
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace HybridSearchAsync through HybridSearchOptions and the Elasticsearch connector to locate how hybrid queries are built. Review the existing option mapping and determine how collapse and inner_hits could be represented without assuming an extension point. Done means the requested field-collapse behavior is supported and verified for chunked records sharing a documentId.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, elasticsearch
- Domain
- backend-api-design, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100