apache / apache/ignite

[IEP] Enhance TextQuery for Hybrid Search: Integrate Keyword and Vector Retrieval using Lucene9+

Open
#13,526 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

### Proposal
Extend the existing `TextQuery` API to support hybrid search, combining Lucene-based keyword matching with vector-based Approximate Nearest Neighbor (ANN) search. This aligns with the vision of IEP-141 to enable "hybrid queries, where filtering by metadata (structured data) and ranking by vector happen in a single execution layer" [citation:2][citation:3].

This approach reuses the familiar `TextQuery` interface, making it intuitive for existing users and avoiding API fragmentation.

### Motivation & Design Goal
- **Unified Query Experience**: Users can perform a search that considers both exact keyword relevance and semantic similarity in one query.
- **Leverage Lucene**: Reuse the existing `GridLuceneIndex` infrastructure to manage both inverted indexes (for text) and vector indexes (e.g., Lucene's `KnnFloatVectorQuery`) [citation:11].
- **Single Execution Layer**: Perform filtering by structured data and ranking by vector similarity within the same query engine, optimizing complex AI workloads like RAG and semantic search [citation:2][citation:3].

### Proposed Interface: Enhanced TextQuery

The `TextQuery` class will be enhanced with new builder-style methods to accept a vector query component.

```java
package org.apache.ignite.cache.query;

/**
* A hybrid query that performs both text and vector search.
* By default, it behaves as a standard TextQuery. When a vector is provided,
* it performs a hybrid search combining Lucene's keyword and KNN queries.
*/
public final class TextQuery extends Query> {
/**
* Standard constructor for pure text search.
*/
public TextQuery(Class type, String txt) { ... }

/**
* NEW: Sets the vector field and query vector for hybrid search.
*
* @param vectorFieldName Name of the vector field (annotated with @QueryVectorField).
* @param vector The query vector (float array).
* @param k Number of nearest neighbors to return from the vector portion.
* @return this TextQuery instance for chaining.
*/
public TextQuery setVectorQuery(String vectorFieldName, float[] vector, int k) { ... }

/**
* NEW: Sets the distance metric for the vector part (defaults to COSINE).
*/
public TextQuery setDistanceMetric(DistanceMetric metric) { ... }

/**
* NEW: Sets the hybrid ranking strategy.
* - RRF: Reciprocal Rank Fusion (default).
* - WEIGHTED_SUM: Weighted sum of text and vector scores.
*/
public TextQuery setHybridStrategy(HybridStrategy strategy) { ... }

/**
* NEW: Sets the weight for the vector portion in a WEIGHTED_SUM strategy.
*/
public TextQuery setVectorWeight(float weight) { ... }

// Existing methods: setPageSize, setLocal, etc. remain unchanged [citation:1][citation:4][citation:5].
}
```

### Proposed Annotation: @QueryVectorField
Same as previously proposed. This annotation marks a field for vector indexing, specifying dimension, data type (FP32, INT8, etc.), and the distance metric .
```
java
package org.apache.ignite.cache.query.annotations;

public @interface QueryVectorField {
int dimension();
VectorDataType dataType() default VectorDataType.FP32;
DistanceMetric metric() default DistanceMetric.COSINE;
}
```

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing TextQuery API and the GridLuceneIndex infrastructure named in the proposal, then review the related query annotation package. The work is done when TextQuery supports the proposed vector inputs, distance metric, and hybrid ranking strategies while preserving standard text-query behavior; the payload does not name tests or implementation files beyond these entry points.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, databases, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.