agentscope-ai / agentscope-ai/agentscope-java

[Bug]: WordReader collapses paragraph boundaries into a single \n, making SplitStrategy.PARAGRAPH silently degrade to character chunking

Abierto
#2,964 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Java
Estrellas
5.6k
Forks
1.3k
Merge medio
4 d 12 h
PR fusionados (30 d)
77

Descripción

**Describe the bug**

`WordReader.getDataBlocks()` joins consecutive paragraphs with a single `"\n"` and skips empty paragraphs entirely.

In WordprocessingML every `` is a paragraph in its own right, and a blank line is an empty `` element with no `` child. Pressing Enter twice in Word produces:

```xml
Paragraph one.
...
Paragraph two.
```

For the middle element `extractTextFromParagraph()` returns `""` (there are no runs, so both the `para.getText()` path and the run-concatenation fallback yield an empty string). The `if (text != null && !text.isEmpty())` guard then skips the whole branch — no characters are written and no flag is set. **The fact that a paragraph existed here is destroyed at that point and cannot be recovered downstream.**

The result is `"Paragraph one.\nParagraph two."`: the blank line is gone, and a paragraph boundary has become indistinguishable from an in-paragraph line break.

**Why this matters beyond formatting**

`TextChunker` recognises paragraph boundaries only via:

```java
private static final Pattern PARAGRAPH_SEPARATOR = Pattern.compile("\\n\\s*\\n");
```

This requires **two** newlines. Since `getDataBlocks()` never emits `\n\n`, `PARAGRAPH_SEPARATOR.split(text)` always returns a single element covering the whole document. `chunkByParagraph()` therefore loops once, inevitably falls into the `currentChunk.length() > chunkSize` branch, and delegates to `chunkByCharacter()`.

In other words, **`SplitStrategy.PARAGRAPH` — the default for `WordReader` — has never actually performed paragraph-aware chunking.** The degradation is silent: no exception, no warning, and a non-empty chunk list is still returned.

**To Reproduce**

```java
List docs = new WordReader() // defaults: chunkSize=512, PARAGRAPH, overlap=50
.read(ReaderInput.fromPath(Paths.get("your.docx")))
.block();
for (Document d : docs) {
System.out.println(d.getMetadata().getContentText().length());
}
```

Measured on two real-world documents (Chinese corporate policy documents, 53 and 36 body paragraphs):

| Document | `\n\s*\n` matches | chunk sizes | sample cut point |
|---|---|---|---|
| A (53 paragraphs, 8 blank, 1 table) | 1 | 512 / 512 / 512 / 286 | mid-token |
| B (36 paragraphs, 0 blank) | 1 | 512 / 512 / 432 | mid-sentence |

With the paragraph separator changed to `"\n\n"`:

| Document | `\n\s*\n` matches | chunk sizes |
|---|---|---|
| A | 46 (= 45 non-empty paragraphs + 1 table) | 506 / 509 / 495 / 357 |
| B | 36 | 512 / 498 / 481 |

**Expected behavior**

A `` boundary should be encoded as a blank line (`\n\n`), so that the paragraph structure of the source document survives into the chunking stage and `SplitStrategy.PARAGRAPH` actually takes effect.

**Error messages**

None. No exception is thrown — that is precisely the problem: the degradation is entirely silent.

**Environment**

- AgentScope-Java Version: 2.0.0 (also reproducible on 1.0.12). `WordReader.java` is byte-identical across 1.0.12, 2.0.0 and current `main` (md5 `397960db581a057292110d352b40bab7`, 20557 bytes), so `main` is affected as well
- Java Version: 21
- OS: macOS

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.