agentscope-ai / agentscope-ai/ReMe

[Bug]: Persisted chunk store is not invalidated when the chunker changes — stale "full-outline" chunks survive upgrades and are never re-chunked

Ouverte
#524 1 commentaire 0 réactions 1 personne assignée Assignée à @ployts Voir sur GitHub
bug
Langage dominant
Python
Étoiles
3.4k
Forks
298
Merge moyen
19 h 52 min
PR mergées (30 j)
55

Description

### Description

## Observed

An older MarkdownFileChunker version embedded the COMPLETE document outline (every heading
in the file) into every chunk it produced — a form of quadratic outline duplication. That
generation bug is already fixed in the current code (the `MarkdownFileChunker` docstring now
explicitly says it avoids "the quadratic growth caused by repeating the complete document
outline in every chunk"), and re-running the current chunker on the same file yields clean,
small chunks.

However, the persisted chunk store is never invalidated when the chunker changes:

- `LocalFileStore.store_version` is a static `"v1"` string baked into the store filename
(`file_chunks_{name}_{store_version}.jsonl.zst`); it is not tied to the chunker logic.
- After upgrading ReMe, files whose mtime did not change are NOT re-chunked: the old
(poisoned) chunks are loaded as-is from the store.
- `_repair_graph_chunk_consistency` cannot help: it only detects a graph<->store mismatch.
Here both the file graph and the chunk store consistently reference the SAME old chunk ids,
so no mismatch is detected and no automatic re-chunk is triggered.

Concrete impact observed on a ~20KB / 13-section markdown memory file: the persisted store
contained **13 chunks**, each carrying the full document outline (all 13 section headings)
plus one section's body — 9 of them looked nearly identical when only head/tail are shown
(they share the same outline prefix/suffix). The current chunker re-chunks the same file into
**3 clean chunks**. Across the workspace, 513 stored chunks for 378 files should be ~492 with
the current chunker.

When the user later switches the embedding model (vector-space change) and runs a full
re-embed, all these stale/duplicated chunks get re-embedded, wasting tokens/CPU and polluting
retrieval (the same file surfaces many times in search results with near-identical content).

## Expected

The chunk store should be invalidated when the chunking logic changes:

- Include a chunker fingerprint (hash of the chunker class + its configuration, e.g.
`chunk_byte_size`, `embed_toc`, `max_ast_sections`, AST/plain-text mode) in the store
metadata or filename (bump `store_version`).
- On load, if the fingerprint differs from the current chunker, treat the store as stale:
drop persisted chunks for changed files (or all) and re-chunk from source, so upgraded
deployments self-heal instead of silently keeping poisoned chunks.

### Steps to reproduce

1. On an older ReMe version whose markdown chunker embedded the full outline in every chunk,
index a markdown file with multiple `##`/`###` sections (each section becomes one chunk
that repeats the whole heading list). Persist the store.
2. Upgrade ReMe to a version with the fixed chunker. Do NOT modify the source files (mtime
unchanged).
3. Restart: the store is loaded as-is; the file still has N chunks, each containing the full
document outline. No re-chunk happens, `_repair_graph_chunk_consistency` reports no
mismatch.
4. (Optional) Switch embedding model and rebuild the embedding index: all the stale
duplicated chunks are re-embedded.

### Relevant configuration

```yaml
- file store backend: LocalFileStore
- chunker: MarkdownFileChunker (default: chunk_byte_size=10000, embed_toc=True,
max_ast_sections=100)
- store_version: "v1"
```

### Logs or traceback

```shell
# store loads the stale poisoned chunks without re-chunking:
INFO reme.components.file_store.local_file_store | Loaded 513 chunks from .../file_chunks_default_v1.jsonl.zst
INFO reme.components.file_store.local_file_store | default: chunk store load complete: chunks=513

# graph/store are consistent, so no automatic repair triggers:
INFO reme.steps.index.init_changes | [InitChangesStep] file_store:default is up to date

(no secrets in these logs)
```

### ReMe version

0.4.1.10

### Python version

3.11.9

### Operating system

Linux

### Affected area

Other

### Data safety

- [x] I removed credentials and private memory content from this report.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.