apache / apache/geaflow

[enhancement] Implement multi-channel retrieval, graph expansion, and fusion execution

Open
#865 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
808
Forks
188
Avg merge
3d 22h
Merged PRs (30d)
2

Description

**Phase 2 of Parent epic:** [#863](https://github.com/apache/geaflow/issues/863)

## Summary

Build the retrieval core over the versioned graph and indexes delivered by Phase 1. The core must expose BM25, vector, and entity-anchor channels through one Java API, expand selected anchors by at most two hops under explicit budgets, and return candidates through a common evidence protocol.

This phase also adds deterministic deduplication, fusion, truncation, and sequential/parallel/ cascaded execution. Adaptive policy selection is added in Phase 3, but this phase must provide the executor and plan interfaces that the policy can call later.

## Motivation

The existing memory search path is session-oriented and loses channel scores during execution. A shared retrieval core is needed before different modes can be compared fairly or called by a production service.

## Scope

- Implement `TextRetriever`/Lucene BM25 retrieval with real scores, stable ranks, field mappings, `topK`, and candidate limits.
- Implement a replaceable `VectorRetriever` that accepts precomputed vectors and does not depend on a particular embedding service.
- Implement `EntityAnchorResolver` using normalized names, aliases, and entities attached to highly ranked chunks. Graph-only mode must return a structured empty result when no reliable anchor exists.
- Implement `GraphRetriever` with one- and two-hop expansion, lazy edge scanning, edge-scan caps, neighbor sampling, relation-type priority, deterministic sampling, and diffusion stopping rules.
- Convert every channel result to the unified `Evidence` model while preserving raw channel scores, ranks, source/chunk identity, and graph paths.
- Deduplicate first by stable evidence ID and then by normalized source/chunk identity.
- Implement weighted reciprocal-rank fusion by default, an optional reranker interface, and final `topK`/token-budget truncation.
- Implement sequential, parallel, and cascaded executors with per-stage budget accounting.
- Support BM25-only, Vector-only, Graph-only, and Fixed Hybrid modes. Expose a validated plan interface for Adaptive Hybrid without embedding routing rules in channel implementations.

## Suggested implementation locations

- `geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/channel/` - BM25, vector, and anchor retrievers.
- `geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/graph/` - neighbor provider, frontier, path, sampling, and stopping logic.
- `geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/fusion/` - evidence merger, RRF, reranker SPI, and token truncation.
- `geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/execution/` - stage executor and execution modes.
- `geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/` - channel, graph, fusion, and budget tests using the fixed fixture from Phase 1.

## Testing requirements

- Unit tests for Lucene score preservation, topK, candidate caps, vector ordering, anchor confidence, path scoring, deterministic sampling, and diffusion stopping.
- Fusion tests for duplicate removal, score/rank preservation, RRF ordering, and token truncation.
- Concurrency tests for parallel execution and isolation of per-request budget/trace state.
- End-to-end tests against the Phase 1 in-memory fixture for all four fixed modes and all execution modes.

## Dependencies and follow-up

- Keep channel and executor interfaces independent of Solon so they can also be used by offline benchmark tooling.

Contributor guide

Open the contributing guide

Research direction

Start by reading the Phase 1 fixture and the suggested retrieval packages: channel, graph, fusion, and execution. Run the retrieval tests under geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/; done means the four fixed modes and execution modes satisfy the listed scoring, fusion, budget, concurrency, and end-to-end tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.