spring-projects / spring-projects/spring-ai
After updating the Document in spring ai core, I encountered the following problem
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 6
Description
This is my code:
Resource resource = new FileSystemResource(filePath);
List<Document> documents = new TikaDocumentReader(resource).read();
return new TokenTextSplitter(knowledgeBaseFileSlice.getDefaultChunkSize(),
knowledgeBaseFileSlice.getMinChunkSizeChars(),
knowledgeBaseFileSlice.getMinChunkLengthToEmbed(), knowledgeBaseFileSlice.getMaxNumChunks(),
knowledgeBaseFileSlice.isKeepSeparator()).apply(documents);
After TikaDocumentReader reads a Word document, the content read not only includes the text of the document, but also the XML information of the file, if I use getText(), the output will include the following content, like this:
docProps/app.xml
Normal.dotm 1 0 0 0 0 0 false false 0 WPS Office_10.1.0.7698_F1E327BC-269C-435d-A152-05C5408002CA 0
docProps/core.xml
2023-08-26T18:18:00Z admin admin 2023-08-26T18:18:41Z 1
docProps/custom.xml
2052-10.1.0.7698
word/styles.xml
word/settings.xml
word/theme/theme1.xml
word/document.xml
What went wrong???
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the behavior with TikaDocumentReader on the reported Word document and inspect the value returned by getText() before TokenTextSplitter is applied. Compare the extracted text with the document contents and determine whether the XML entries are expected; done means the cause and an appropriate extraction behavior are clearly established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100