elastic / elastic/ai-dotnet

Support Elasticsearch field collapse (deduplication by documentId) in HybridSearch

Open
#20 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
21
Forks
4
Avg merge
1h 18m
Merged PRs (30d)
1

Description

**Description**
We are using Microsoft.Extensions.VectorData with Elasticsearch as the vector store and calling:

`IKeywordHybridSearchable.HybridSearchAsync(...)`

Our data is chunked (multiple records per logical document). Each chunk shares the same documentId.

During hybrid search, multiple chunks from the same document are returned, which results in duplicate documents at the application level. Elasticsearch natively supports solving this using the collapse parameter:
```

"collapse": {
"field": "documentId",
"inner_hits": {
"size": 1,
"sort": [{ "_score": "desc" }]
}
}
```

However, HybridSearchOptions currently exposes only:

1. Filter
2. VectorProperty
3. AdditionalProperty
4. Skip
5. IncludeVectors

There is no way to specify collapse or grouping options, and collapse cannot be expressed via Filter.

**Problem Statement**

When using chunked indexing:
Hybrid search returns multiple chunks from the same document
Consumers must manually deduplicate results
This is inefficient and differs from Elasticsearch-native behavior
We need a way to request one result per logical document directly at the Elasticsearch query

**Proposed Enhancement**

Add optional collapse support to HybridSearchOptions, for example:

```
public class HybridSearchOptions
{
...
public string? CollapseField { get; set; } // e.g. "documentId"
public int? CollapseInnerHitsSize { get; set; } = 1;
}

```

Then, in the Elasticsearch connector, map this to:
```

"collapse": {
"field": "",
"inner_hits": {
"size":
}
}
```

**Benefits**

Prevents duplicate documents when using chunked storage
Aligns hybrid search behavior with native Elasticsearch capabilities
Improves relevance, performance, and RAG evidence quality
Removes need for application-side deduplication

**Could you please advise where in the Elasticsearch connector/library we should add support for the collapse parameter (or an equivalent option) for HybridSearchAsync, or if there is an existing extension point or recommended approach to implement Elasticsearch field collapse (e.g., by documentId) without forking the entire library?**

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace HybridSearchAsync through HybridSearchOptions and the Elasticsearch connector to locate how hybrid queries are built. Review the existing option mapping and determine how collapse and inner_hits could be represented without assuming an extension point. Done means the requested field-collapse behavior is supported and verified for chunked records sharing a documentId.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, elasticsearch
Domain
backend-api-design, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.