spring-projects / spring-projects/spring-ai

After updating the Document in spring ai core, I encountered the following problem

Open
#1,968 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

This is my code:

   Resource resource = new FileSystemResource(filePath);
    List<Document> documents = new TikaDocumentReader(resource).read();
    return new TokenTextSplitter(knowledgeBaseFileSlice.getDefaultChunkSize(), 
            knowledgeBaseFileSlice.getMinChunkSizeChars(),
            knowledgeBaseFileSlice.getMinChunkLengthToEmbed(), knowledgeBaseFileSlice.getMaxNumChunks(),
            knowledgeBaseFileSlice.isKeepSeparator()).apply(documents);

After TikaDocumentReader reads a Word document, the content read not only includes the text of the document, but also the XML information of the file, if I use getText(), the output will include the following content, like this:
docProps/app.xml
Normal.dotm 1 0 0 0 0 0 false false 0 WPS Office_10.1.0.7698_F1E327BC-269C-435d-A152-05C5408002CA 0

docProps/core.xml
2023-08-26T18:18:00Z admin admin 2023-08-26T18:18:41Z 1

docProps/custom.xml
2052-10.1.0.7698

word/styles.xml

word/settings.xml

word/theme/theme1.xml

word/document.xml

What went wrong???

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the behavior with TikaDocumentReader on the reported Word document and inspect the value returned by getText() before TokenTextSplitter is applied. Compare the extracted text with the document contents and determine whether the XML entries are expected; done means the cause and an appropriate extraction behavior are clearly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.