agentscope-ai / agentscope-ai/agentscope-java

[Bug]: WordReader collapses paragraph boundaries into a single \n, making SplitStrategy.PARAGRAPH silently degrade to character chunking

Aberta
#2,964 0 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Java
Estrelas
5.6k
Forks
1.3k
Merge médio
4d 12h
PRs com merge (30d)
77

Descrição

**Describe the bug**

`WordReader.getDataBlocks()` joins consecutive paragraphs with a single `"\n"` and skips empty paragraphs entirely.

In WordprocessingML every `` is a paragraph in its own right, and a blank line is an empty `` element with no `` child. Pressing Enter twice in Word produces:

```xml
Paragraph one.
...
Paragraph two.
```

For the middle element `extractTextFromParagraph()` returns `""` (there are no runs, so both the `para.getText()` path and the run-concatenation fallback yield an empty string). The `if (text != null && !text.isEmpty())` guard then skips the whole branch — no characters are written and no flag is set. **The fact that a paragraph existed here is destroyed at that point and cannot be recovered downstream.**

The result is `"Paragraph one.\nParagraph two."`: the blank line is gone, and a paragraph boundary has become indistinguishable from an in-paragraph line break.

**Why this matters beyond formatting**

`TextChunker` recognises paragraph boundaries only via:

```java
private static final Pattern PARAGRAPH_SEPARATOR = Pattern.compile("\\n\\s*\\n");
```

This requires **two** newlines. Since `getDataBlocks()` never emits `\n\n`, `PARAGRAPH_SEPARATOR.split(text)` always returns a single element covering the whole document. `chunkByParagraph()` therefore loops once, inevitably falls into the `currentChunk.length() > chunkSize` branch, and delegates to `chunkByCharacter()`.

In other words, **`SplitStrategy.PARAGRAPH` — the default for `WordReader` — has never actually performed paragraph-aware chunking.** The degradation is silent: no exception, no warning, and a non-empty chunk list is still returned.

**To Reproduce**

```java
List docs = new WordReader() // defaults: chunkSize=512, PARAGRAPH, overlap=50
.read(ReaderInput.fromPath(Paths.get("your.docx")))
.block();
for (Document d : docs) {
System.out.println(d.getMetadata().getContentText().length());
}
```

Measured on two real-world documents (Chinese corporate policy documents, 53 and 36 body paragraphs):

| Document | `\n\s*\n` matches | chunk sizes | sample cut point |
|---|---|---|---|
| A (53 paragraphs, 8 blank, 1 table) | 1 | 512 / 512 / 512 / 286 | mid-token |
| B (36 paragraphs, 0 blank) | 1 | 512 / 512 / 432 | mid-sentence |

With the paragraph separator changed to `"\n\n"`:

| Document | `\n\s*\n` matches | chunk sizes |
|---|---|---|
| A | 46 (= 45 non-empty paragraphs + 1 table) | 506 / 509 / 495 / 357 |
| B | 36 | 512 / 498 / 481 |

**Expected behavior**

A `` boundary should be encoded as a blank line (`\n\n`), so that the paragraph structure of the source document survives into the chunking stage and `SplitStrategy.PARAGRAPH` actually takes effect.

**Error messages**

None. No exception is thrown — that is precisely the problem: the degradation is entirely silent.

**Environment**

- AgentScope-Java Version: 2.0.0 (also reproducible on 1.0.12). `WordReader.java` is byte-identical across 1.0.12, 2.0.0 and current `main` (md5 `397960db581a057292110d352b40bab7`, 20557 bytes), so `main` is affected as well
- Java Version: 21
- OS: macOS

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.