agentscope-ai / agentscope-ai/ReMe
[Bug]: Persisted chunk store is not invalidated when the chunker changes — stale "full-outline" chunks survive upgrades and are never re-chunked
- 主要語言
- Python
- 星號
- 3.5k
- 分支
- 298
- 平均合併
- 19 小時 52 分鐘
- 30 天內合併 PR
- 55
描述
### Description
## Observed
An older MarkdownFileChunker version embedded the COMPLETE document outline (every heading
in the file) into every chunk it produced — a form of quadratic outline duplication. That
generation bug is already fixed in the current code (the `MarkdownFileChunker` docstring now
explicitly says it avoids "the quadratic growth caused by repeating the complete document
outline in every chunk"), and re-running the current chunker on the same file yields clean,
small chunks.
However, the persisted chunk store is never invalidated when the chunker changes:
- `LocalFileStore.store_version` is a static `"v1"` string baked into the store filename
(`file_chunks_{name}_{store_version}.jsonl.zst`); it is not tied to the chunker logic.
- After upgrading ReMe, files whose mtime did not change are NOT re-chunked: the old
(poisoned) chunks are loaded as-is from the store.
- `_repair_graph_chunk_consistency` cannot help: it only detects a graph<->store mismatch.
Here both the file graph and the chunk store consistently reference the SAME old chunk ids,
so no mismatch is detected and no automatic re-chunk is triggered.
Concrete impact observed on a ~20KB / 13-section markdown memory file: the persisted store
contained **13 chunks**, each carrying the full document outline (all 13 section headings)
plus one section's body — 9 of them looked nearly identical when only head/tail are shown
(they share the same outline prefix/suffix). The current chunker re-chunks the same file into
**3 clean chunks**. Across the workspace, 513 stored chunks for 378 files should be ~492 with
the current chunker.
When the user later switches the embedding model (vector-space change) and runs a full
re-embed, all these stale/duplicated chunks get re-embedded, wasting tokens/CPU and polluting
retrieval (the same file surfaces many times in search results with near-identical content).
## Expected
The chunk store should be invalidated when the chunking logic changes:
- Include a chunker fingerprint (hash of the chunker class + its configuration, e.g.
`chunk_byte_size`, `embed_toc`, `max_ast_sections`, AST/plain-text mode) in the store
metadata or filename (bump `store_version`).
- On load, if the fingerprint differs from the current chunker, treat the store as stale:
drop persisted chunks for changed files (or all) and re-chunk from source, so upgraded
deployments self-heal instead of silently keeping poisoned chunks.
### Steps to reproduce
1. On an older ReMe version whose markdown chunker embedded the full outline in every chunk,
index a markdown file with multiple `##`/`###` sections (each section becomes one chunk
that repeats the whole heading list). Persist the store.
2. Upgrade ReMe to a version with the fixed chunker. Do NOT modify the source files (mtime
unchanged).
3. Restart: the store is loaded as-is; the file still has N chunks, each containing the full
document outline. No re-chunk happens, `_repair_graph_chunk_consistency` reports no
mismatch.
4. (Optional) Switch embedding model and rebuild the embedding index: all the stale
duplicated chunks are re-embedded.
### Relevant configuration
```yaml
- file store backend: LocalFileStore
- chunker: MarkdownFileChunker (default: chunk_byte_size=10000, embed_toc=True,
max_ast_sections=100)
- store_version: "v1"
```
### Logs or traceback
```shell
# store loads the stale poisoned chunks without re-chunking:
INFO reme.components.file_store.local_file_store | Loaded 513 chunks from .../file_chunks_default_v1.jsonl.zst
INFO reme.components.file_store.local_file_store | default: chunk store load complete: chunks=513
# graph/store are consistent, so no automatic repair triggers:
INFO reme.steps.index.init_changes | [InitChangesStep] file_store:default is up to date
(no secrets in these logs)
```
### ReMe version
0.4.1.10
### Python version
3.11.9
### Operating system
Linux
### Affected area
Other
### Data safety
- [x] I removed credentials and private memory content from this report.
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
Read `LocalFileStore` where `store_version` is used in `file_chunks_{name}_{store_version}.jsonl.zst`, then trace startup load flow for logs like `Loaded ... chunks` and `file_store:default is up to date`. Next inspect how `_repair_graph_chunk_consistency` is called (from `InitChangesStep`) and how it avoids re-embedding when IDs match. Then review `MarkdownFileChunker` config fields (`chunk_byte_size`, `embed_toc`, `max_ast_sections`, AST/plain-text mode) to compare a chunker fingerprint on load; done when stale stores are dropped/rebuilt so upgraded files re-chunk instead of reusing duplicated old chunks.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python
- 領域
- backend
- Issue 類型
- 缺陷
- 難度
- 3/5
- 預估耗時
- 1-2 天
- 活躍度
- 活躍
- 描述清晰度
- 基本清楚
- 新手友好度
- 55/100