agentscope-ai / agentscope-ai/agentscope-java
[Bug]: WordReader collapses paragraph boundaries into a single \n, making SplitStrategy.PARAGRAPH silently degrade to character chunking
- 主要语言
- Java
- 星标
- 5.6k
- 派生
- 1.3k
- 平均合并
- 4 天 12 小时
- 30 天内合并 PR
- 77
描述
**Describe the bug**
`WordReader.getDataBlocks()` joins consecutive paragraphs with a single `"\n"` and skips empty paragraphs entirely.
In WordprocessingML every `` is a paragraph in its own right, and a blank line is an empty `` element with no `` child. Pressing Enter twice in Word produces:
```xml
Paragraph one.
...
Paragraph two.
```
For the middle element `extractTextFromParagraph()` returns `""` (there are no runs, so both the `para.getText()` path and the run-concatenation fallback yield an empty string). The `if (text != null && !text.isEmpty())` guard then skips the whole branch — no characters are written and no flag is set. **The fact that a paragraph existed here is destroyed at that point and cannot be recovered downstream.**
The result is `"Paragraph one.\nParagraph two."`: the blank line is gone, and a paragraph boundary has become indistinguishable from an in-paragraph line break.
**Why this matters beyond formatting**
`TextChunker` recognises paragraph boundaries only via:
```java
private static final Pattern PARAGRAPH_SEPARATOR = Pattern.compile("\\n\\s*\\n");
```
This requires **two** newlines. Since `getDataBlocks()` never emits `\n\n`, `PARAGRAPH_SEPARATOR.split(text)` always returns a single element covering the whole document. `chunkByParagraph()` therefore loops once, inevitably falls into the `currentChunk.length() > chunkSize` branch, and delegates to `chunkByCharacter()`.
In other words, **`SplitStrategy.PARAGRAPH` — the default for `WordReader` — has never actually performed paragraph-aware chunking.** The degradation is silent: no exception, no warning, and a non-empty chunk list is still returned.
**To Reproduce**
```java
List docs = new WordReader() // defaults: chunkSize=512, PARAGRAPH, overlap=50
.read(ReaderInput.fromPath(Paths.get("your.docx")))
.block();
for (Document d : docs) {
System.out.println(d.getMetadata().getContentText().length());
}
```
Measured on two real-world documents (Chinese corporate policy documents, 53 and 36 body paragraphs):
| Document | `\n\s*\n` matches | chunk sizes | sample cut point |
|---|---|---|---|
| A (53 paragraphs, 8 blank, 1 table) | 1 | 512 / 512 / 512 / 286 | mid-token |
| B (36 paragraphs, 0 blank) | 1 | 512 / 512 / 432 | mid-sentence |
With the paragraph separator changed to `"\n\n"`:
| Document | `\n\s*\n` matches | chunk sizes |
|---|---|---|
| A | 46 (= 45 non-empty paragraphs + 1 table) | 506 / 509 / 495 / 357 |
| B | 36 | 512 / 498 / 481 |
**Expected behavior**
A `` boundary should be encoded as a blank line (`\n\n`), so that the paragraph structure of the source document survives into the chunking stage and `SplitStrategy.PARAGRAPH` actually takes effect.
**Error messages**
None. No exception is thrown — that is precisely the problem: the degradation is entirely silent.
**Environment**
- AgentScope-Java Version: 2.0.0 (also reproducible on 1.0.12). `WordReader.java` is byte-identical across 1.0.12, 2.0.0 and current `main` (md5 `397960db581a057292110d352b40bab7`, 20557 bytes), so `main` is affected as well
- Java Version: 21
- OS: macOS
贡献指南
评估
这个 Issue 还没有评估数据。