[geaflow/ai-memory] Implement deterministic `Chunker` SPI
- Dominant language
- Java
- Stars
- 808
- Forks
- 188
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 2
Description
Priority: P0
Difficulty: Intermediate
Context: `ModelUtils.splitLongText` can split long strings, but it does not create stable chunk IDs, offsets, source references, or replay semantics. Chunk identity is required for chunk vector retrieval and citation-aware answers.
Scope:
- Add `Chunker` interface and a default text chunker.
- Add `ChunkRecord` with `chunk_id`, `source_ref`, `ordinal`, `start_offset`, `end_offset`, `text_hash`, `policy_version`.
- Add golden tests for Chinese, English, mixed punctuation, CRLF, blank lines, and long paragraphs.
Constraints:
- Do not call embedding services.
- Do not decide extraction schema in this issue.
- Chunk IDs must not depend on wall-clock time.
Acceptance Criteria:
- Same document + same chunk policy produces identical chunk IDs across runs.
- Changing chunk policy changes `policy_version` and expected chunk IDs.
- Empty input produces an explicit empty result, not null.
Suggested paths:
- `geaflow-ai/src/main/java/org/apache/geaflow/ai/chunk`
- `geaflow-ai/src/test/resources/chunk`
Contributor guide
Research direction
Start by reading ModelUtils.splitLongText and the suggested geaflow-ai/src/main/java/org/apache/geaflow/ai/chunk package. Use geaflow-ai/src/test/resources/chunk for golden cases covering the listed inputs. Done means the Chunker, default text chunker, and ChunkRecord satisfy deterministic IDs, policy-version changes, and an explicit empty result without embedding calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ai, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100