apache / apache/geaflow

[Geaflow/ai-memory] Define `DocumentSource` and `DocumentRecord` contracts

Open
#833 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
808
Forks
188
Avg merge
3d 22h
Merged PRs (30d)
2

Description

Priority: P0
Difficulty: Starter
Suggested labels: `area:ai-memory`, `type:contract`, `difficulty:starter`

Context: `TextFileReader` currently reads non-empty lines into strings. Graph Memory Phase 1 needs a stable document boundary before chunking, extraction, indexing, and replay can be deterministic.

Scope:

- Add contract classes for `DocumentSource`, `DocumentRecord`, and `SourceRef`.
- Include fields such as `source_id`, `uri`, `media_type`, `charset`, `content_hash`, `created_at`, `ingested_at`, and `metadata`.
- Add tests for empty files, duplicate source IDs, Unicode content, and stable content hash.

Constraints:

- Do not replace all existing readers in this issue.
- Do not add remote storage support.
- Do not store raw private text in logs.

Acceptance Criteria:

- A document loaded twice with the same content receives the same `content_hash`.
- Invalid or missing source metadata fails with a typed validation error.
- Existing `TextFileReader` tests remain compatible.

Suggested paths:

- `geaflow-ai/src/main/java/org/apache/geaflow/ai/graph/io`
- `geaflow-ai/src/test/java/org/apache/geaflow/ai`

Contributor guide

Open the contributing guide

Research direction

Start in geaflow-ai/src/main/java/org/apache/geaflow/ai/graph/io and inspect the existing TextFileReader and its tests under geaflow-ai/src/test/java/org/apache/geaflow/ai. Define the DocumentSource, DocumentRecord, and SourceRef contracts with validation and stable hashing, then add coverage for empty files, duplicate source IDs, Unicode content, and repeated loads. Done means the acceptance criteria pass and existing TextFileReader tests remain compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.