spring-projects / spring-projects/spring-ai

TextSplitter loses Document score and parent tracking information during splitting

Open
#4,428 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Expected Behavior

When splitting documents using TextSplitter, all original document properties should be preserved in the resulting chunks, and users should be able to track the relationship between chunks and their parent documents. This is essential for RAG (Retrieval-Augmented Generation) systems that need to:

  1. Maintain document relevance scores across chunks
  2. Reconstruct original documents from chunks
  3. Group search results by source document
  4. Provide proper attribution and traceability

Users should be able to:

//Split a document with score and metadata
Document originalDoc = Document.builder()
    .text("Long document content...")
    .score(0.95)
    .metadata(Map.of("source", "report.pdf", "author": "John Doe"))
    .build();

List<Document> chunks = textSplitter.split(originalDoc);

//Access preserved score
chunks.get(0).getScore(); // Should return 0.95

//Track parent document
String parentId = (String) chunks.get(0).getMetadata().get("parent_document_id");
int chunkIndex = (Integer) chunks.get(0).getMetadata().get("chunk_index");
int totalChunks = (Integer) chunks.get(0).getMetadata().get("total_chunks");

//Reconstruct document order
chunks.stream()
    .filter(chunk -> parentId.equals(chunk.getMetadata().get("parent_document_id")))
    .sorted((a, b) -> Integer.compare(
        (Integer) a.getMetadata().get("chunk_index"),
        (Integer) b.getMetadata().get("chunk_index")
    ));

Current Behavior

The current TextSplitter implementation has significant limitations that impact RAG system functionality:

  1. Property Loss: Document score values are completely lost during splitting, making it impossible to maintain relevance rankings
  2. Missing Traceability: There's no way to determine which original document a chunk came from, breaking document attribution
  3. No Chunk Context: Users cannot determine chunk position or total count, making document reconstruction impossible
  4. Incomplete Implementation: The TODO comment "copy over other properties" indicates known missing functionality

Current behavior results in:

Document originalDoc = Document.builder()
    .text("Content...")
    .score(0.95)  // This score is lost
    .build();

List<Document> chunks = textSplitter.split(originalDoc);
chunks.get(0).getScore(); // Returns null instead of 0.95
chunks.get(0).getMetadata(); // Missing parent tracking information

This forces developers to implement workarounds like:

  • Manually tracking document relationships in external data structures
  • Re-implementing scoring logic after splitting
  • Using complex metadata schemes to maintain document context
  • Building custom chunk management systems

Impact

This limitation significantly reduces the effectiveness of RAG systems because:

  • Search Quality: Lost relevance scores mean chunks from high-quality documents aren't prioritized
  • User Experience: Cannot provide proper source attribution or document context
  • System Complexity: Forces developers to build custom tracking mechanisms
  • Data Integrity: Risk of losing important document relationships and metadata

The current implementation essentially treats each chunk as an isolated document, breaking the semantic and contextual relationships that are crucial for effective information retrieval and generation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the TextSplitter implementation and its TODO to trace how Document chunks are created and which properties are currently copied. Compare this with the Document builder and accessors; done means chunks retain the original score, metadata, and parent-tracking fields described in the issue, including chunk order and count.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.