agentscope-ai / agentscope-ai/ReMe

[Bug]: Persisted chunk store is not invalidated when the chunker changes — stale "full-outline" chunks survive upgrades and are never re-chunked

未關閉
#524 1 則留言 0 個 reaction 已指派 1 人 已指派給 @ployts 在 GitHub 檢視
bug
主要語言
Python
星號
3.5k
分支
298
平均合併
19 小時 52 分鐘
30 天內合併 PR
55

描述

### Description

## Observed

An older MarkdownFileChunker version embedded the COMPLETE document outline (every heading
in the file) into every chunk it produced — a form of quadratic outline duplication. That
generation bug is already fixed in the current code (the `MarkdownFileChunker` docstring now
explicitly says it avoids "the quadratic growth caused by repeating the complete document
outline in every chunk"), and re-running the current chunker on the same file yields clean,
small chunks.

However, the persisted chunk store is never invalidated when the chunker changes:

- `LocalFileStore.store_version` is a static `"v1"` string baked into the store filename
(`file_chunks_{name}_{store_version}.jsonl.zst`); it is not tied to the chunker logic.
- After upgrading ReMe, files whose mtime did not change are NOT re-chunked: the old
(poisoned) chunks are loaded as-is from the store.
- `_repair_graph_chunk_consistency` cannot help: it only detects a graph<->store mismatch.
Here both the file graph and the chunk store consistently reference the SAME old chunk ids,
so no mismatch is detected and no automatic re-chunk is triggered.

Concrete impact observed on a ~20KB / 13-section markdown memory file: the persisted store
contained **13 chunks**, each carrying the full document outline (all 13 section headings)
plus one section's body — 9 of them looked nearly identical when only head/tail are shown
(they share the same outline prefix/suffix). The current chunker re-chunks the same file into
**3 clean chunks**. Across the workspace, 513 stored chunks for 378 files should be ~492 with
the current chunker.

When the user later switches the embedding model (vector-space change) and runs a full
re-embed, all these stale/duplicated chunks get re-embedded, wasting tokens/CPU and polluting
retrieval (the same file surfaces many times in search results with near-identical content).

## Expected

The chunk store should be invalidated when the chunking logic changes:

- Include a chunker fingerprint (hash of the chunker class + its configuration, e.g.
`chunk_byte_size`, `embed_toc`, `max_ast_sections`, AST/plain-text mode) in the store
metadata or filename (bump `store_version`).
- On load, if the fingerprint differs from the current chunker, treat the store as stale:
drop persisted chunks for changed files (or all) and re-chunk from source, so upgraded
deployments self-heal instead of silently keeping poisoned chunks.

### Steps to reproduce

1. On an older ReMe version whose markdown chunker embedded the full outline in every chunk,
index a markdown file with multiple `##`/`###` sections (each section becomes one chunk
that repeats the whole heading list). Persist the store.
2. Upgrade ReMe to a version with the fixed chunker. Do NOT modify the source files (mtime
unchanged).
3. Restart: the store is loaded as-is; the file still has N chunks, each containing the full
document outline. No re-chunk happens, `_repair_graph_chunk_consistency` reports no
mismatch.
4. (Optional) Switch embedding model and rebuild the embedding index: all the stale
duplicated chunks are re-embedded.

### Relevant configuration

```yaml
- file store backend: LocalFileStore
- chunker: MarkdownFileChunker (default: chunk_byte_size=10000, embed_toc=True,
max_ast_sections=100)
- store_version: "v1"
```

### Logs or traceback

```shell
# store loads the stale poisoned chunks without re-chunking:
INFO reme.components.file_store.local_file_store | Loaded 513 chunks from .../file_chunks_default_v1.jsonl.zst
INFO reme.components.file_store.local_file_store | default: chunk store load complete: chunks=513

# graph/store are consistent, so no automatic repair triggers:
INFO reme.steps.index.init_changes | [InitChangesStep] file_store:default is up to date

(no secrets in these logs)
```

### ReMe version

0.4.1.10

### Python version

3.11.9

### Operating system

Linux

### Affected area

Other

### Data safety

- [x] I removed credentials and private memory content from this report.

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

Read `LocalFileStore` where `store_version` is used in `file_chunks_{name}_{store_version}.jsonl.zst`, then trace startup load flow for logs like `Loaded ... chunks` and `file_store:default is up to date`. Next inspect how `_repair_graph_chunk_consistency` is called (from `InitChangesStep`) and how it avoids re-embedding when IDs match. Then review `MarkdownFileChunker` config fields (`chunk_byte_size`, `embed_toc`, `max_ast_sections`, AST/plain-text mode) to compare a chunker fingerprint on load; done when stale stores are dropped/rebuilt so upgraded files re-chunk instead of reusing duplicated old chunks.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
backend
Issue 類型
缺陷
難度
3/5
預估耗時
1-2 天
活躍度
活躍
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。