agentscope-ai / agentscope-ai/ReMe
[Bug]: Persisted chunk store is not invalidated when the chunker changes — stale "full-outline" chunks survive upgrades and are never re-chunked
- Langage dominant
- Python
- Étoiles
- 3.4k
- Forks
- 298
- Merge moyen
- 19 h 52 min
- PR mergées (30 j)
- 55
Description
### Description
## Observed
An older MarkdownFileChunker version embedded the COMPLETE document outline (every heading
in the file) into every chunk it produced — a form of quadratic outline duplication. That
generation bug is already fixed in the current code (the `MarkdownFileChunker` docstring now
explicitly says it avoids "the quadratic growth caused by repeating the complete document
outline in every chunk"), and re-running the current chunker on the same file yields clean,
small chunks.
However, the persisted chunk store is never invalidated when the chunker changes:
- `LocalFileStore.store_version` is a static `"v1"` string baked into the store filename
(`file_chunks_{name}_{store_version}.jsonl.zst`); it is not tied to the chunker logic.
- After upgrading ReMe, files whose mtime did not change are NOT re-chunked: the old
(poisoned) chunks are loaded as-is from the store.
- `_repair_graph_chunk_consistency` cannot help: it only detects a graph<->store mismatch.
Here both the file graph and the chunk store consistently reference the SAME old chunk ids,
so no mismatch is detected and no automatic re-chunk is triggered.
Concrete impact observed on a ~20KB / 13-section markdown memory file: the persisted store
contained **13 chunks**, each carrying the full document outline (all 13 section headings)
plus one section's body — 9 of them looked nearly identical when only head/tail are shown
(they share the same outline prefix/suffix). The current chunker re-chunks the same file into
**3 clean chunks**. Across the workspace, 513 stored chunks for 378 files should be ~492 with
the current chunker.
When the user later switches the embedding model (vector-space change) and runs a full
re-embed, all these stale/duplicated chunks get re-embedded, wasting tokens/CPU and polluting
retrieval (the same file surfaces many times in search results with near-identical content).
## Expected
The chunk store should be invalidated when the chunking logic changes:
- Include a chunker fingerprint (hash of the chunker class + its configuration, e.g.
`chunk_byte_size`, `embed_toc`, `max_ast_sections`, AST/plain-text mode) in the store
metadata or filename (bump `store_version`).
- On load, if the fingerprint differs from the current chunker, treat the store as stale:
drop persisted chunks for changed files (or all) and re-chunk from source, so upgraded
deployments self-heal instead of silently keeping poisoned chunks.
### Steps to reproduce
1. On an older ReMe version whose markdown chunker embedded the full outline in every chunk,
index a markdown file with multiple `##`/`###` sections (each section becomes one chunk
that repeats the whole heading list). Persist the store.
2. Upgrade ReMe to a version with the fixed chunker. Do NOT modify the source files (mtime
unchanged).
3. Restart: the store is loaded as-is; the file still has N chunks, each containing the full
document outline. No re-chunk happens, `_repair_graph_chunk_consistency` reports no
mismatch.
4. (Optional) Switch embedding model and rebuild the embedding index: all the stale
duplicated chunks are re-embedded.
### Relevant configuration
```yaml
- file store backend: LocalFileStore
- chunker: MarkdownFileChunker (default: chunk_byte_size=10000, embed_toc=True,
max_ast_sections=100)
- store_version: "v1"
```
### Logs or traceback
```shell
# store loads the stale poisoned chunks without re-chunking:
INFO reme.components.file_store.local_file_store | Loaded 513 chunks from .../file_chunks_default_v1.jsonl.zst
INFO reme.components.file_store.local_file_store | default: chunk store load complete: chunks=513
# graph/store are consistent, so no automatic repair triggers:
INFO reme.steps.index.init_changes | [InitChangesStep] file_store:default is up to date
(no secrets in these logs)
```
### ReMe version
0.4.1.10
### Python version
3.11.9
### Operating system
Linux
### Affected area
Other
### Data safety
- [x] I removed credentials and private memory content from this report.
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.