cloudwego / cloudwego/eino-ext

feat(valkey): add Valkey indexer and retriever components

Open
#879 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
811
Forks
368
Avg merge
16h 22m
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**

Eino currently has no native Valkey retriever or indexer. The existing Redis implementations (components/retriever/redis,
components/indexer/redis) use go-redis/v9, which may incidentally work against a Valkey server but isn't purpose-built for it — it doesn't
leverage the Valkey GLIDE client, can't adopt Valkey Search-specific features cleanly, and introduces go-redis as a transitive dependency
for users who have standardized on Valkey.

Users building RAG pipelines with Valkey as their vector store have no supported path today.

**Describe the solution you'd like**

Add native Valkey Retriever and Indexer components using the valkey-glide-go client:

- components/retriever/valkey/ — implements retriever.Retriever
- components/indexer/valkey/ — implements indexer.Indexer

Retriever capabilities:

- KNN and vector range queries via Valkey Search (FT.SEARCH)
- Hybrid search with filter expressions
- Configurable distance metric, vector field, return fields, dialect
- Embedding integration for automatic query vectorization
- Full Eino callback integration (OnStart, OnEnd, OnError)

Indexer capabilities:

- Batch document storage using pipeline operations (HSET)
- Automatic embedding generation with configurable batch size
- Support for both Hash and JSON document types
- Configurable key prefix and field mapping via DocumentToHashes function
- Full Eino callback integration

Module structure (separate go.mod per package per eino-ext conventions):

components/retriever/valkey/
├── go.mod, retriever.go, retriever_test.go, options.go, consts.go, README.md, examples/
components/indexer/valkey/
├── go.mod, indexer.go, indexer_test.go, consts.go, README.md, examples/

**Describe alternatives you've considered**

1. Adapting existing Redis components — rejected because valkey-glide-go has a different API surface from go-redis/v9. A clean
implementation avoids go-redis as a dependency and can adopt Valkey Search-specific features without backward-compatibility constraints.
2. Separate PRs for retriever and indexer — rejected because upstream convention is to submit both together (qdrant #389, milvus2 #628,
opensearch3 #614). They share client patterns and are useless without each other for a complete RAG pipeline.

**Additional context**

- Client: valkey-glide-go (official Valkey client with native Search command support)
- Backend: Valkey 9.1+ with Search module (valkey/valkey-bundle)
- Reference: components/retriever/redis/ and components/indexer/redis/
- Precedent PRs: qdrant #389, milvus2 #628, opensearch3 #614

Contributor guide

Open the contributing guide

Research direction

Start by reading components/retriever/redis/ and components/indexer/redis/ to understand the existing component patterns, then review valkey-glide-go's Search and pipeline APIs. Implement both components under the specified valkey directories with separate go.mod files, tests, READMEs, and examples. Done means the requested search, indexing, embedding, callback, and document-type capabilities are covered by the tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.