apache / apache/geaflow

[geaflow/ai-memory] Implement deterministic `Chunker` SPI

Open
#834 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
808
Forks
188
Avg merge
3d 22h
Merged PRs (30d)
2

Description

Priority: P0
Difficulty: Intermediate

Context: `ModelUtils.splitLongText` can split long strings, but it does not create stable chunk IDs, offsets, source references, or replay semantics. Chunk identity is required for chunk vector retrieval and citation-aware answers.

Scope:

- Add `Chunker` interface and a default text chunker.
- Add `ChunkRecord` with `chunk_id`, `source_ref`, `ordinal`, `start_offset`, `end_offset`, `text_hash`, `policy_version`.
- Add golden tests for Chinese, English, mixed punctuation, CRLF, blank lines, and long paragraphs.

Constraints:

- Do not call embedding services.
- Do not decide extraction schema in this issue.
- Chunk IDs must not depend on wall-clock time.

Acceptance Criteria:

- Same document + same chunk policy produces identical chunk IDs across runs.
- Changing chunk policy changes `policy_version` and expected chunk IDs.
- Empty input produces an explicit empty result, not null.

Suggested paths:

- `geaflow-ai/src/main/java/org/apache/geaflow/ai/chunk`
- `geaflow-ai/src/test/resources/chunk`

Contributor guide

Open the contributing guide

Research direction

Start by reading ModelUtils.splitLongText and the suggested geaflow-ai/src/main/java/org/apache/geaflow/ai/chunk package. Use geaflow-ai/src/test/resources/chunk for golden cases covering the listed inputs. Done means the Chunker, default text chunker, and ChunkRecord satisfy deterministic IDs, policy-version changes, and an explicit empty result without embedding calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ai, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.