AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[BUG]: Resume preprocessing can duplicate output after crash between checkpoint saves

未关闭
#76 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
bug
主要语言
Python
星标
18
派生
31
平均合并
1 分钟
30 天内合并 PR
2

描述

### Bug Description

The current resume logic in `extract_text_from_xml()` uses `last_processed_page` from `data/checkpoint.json` to decide where to continue, and opens the output file in append mode whenever `start_page > 0`.

However, output is written continuously while checkpoints are only saved every 1000 pages. If the process crashes after writing additional pages but before the next checkpoint update, the output file may already contain more content than the checkpoint reflects.

On resume:
- pages up to `last_processed_page` are skipped
- output file is reopened in append mode
- pages written after the last checkpoint may be appended again

This can duplicate processed content in `data/processed/wiki_clean.txt`, which may also affect processed hashes and Merkle roots.

### Steps to Reproduce

1. Create or use an XML dump with multiple `` entries.
2. Run `extract_text_from_xml()` and let it process beyond the latest saved checkpoint.
3. Simulate an interruption after extra cleaned text has already been written to `data/processed/wiki_clean.txt` but before the next checkpoint update is saved.
4. Ensure `data/checkpoint.json` still contains an older `last_processed_page` value than the content already written to the output file.
5. Run `extract_text_from_xml()` again to resume processing.
6. Observe that pages written after the last checkpoint can be appended again, causing duplicated cleaned text in `data/processed/wiki_clean.txt`.

### Logs and Screenshots

No runtime traceback is required to observe this issue.

The problem is a data-correctness bug in the resume logic:
- checkpoint progress is tracked by page number
- output progress is tracked only by the contents already appended to `wiki_clean.txt`
- if these get out of sync after an interruption, resume may duplicate already-written content

### Environment Details

- OS: Windows 11
- Python version: 3.x
- Repository: AOSSIE-Org/OpenVerifiableLLM
- Relevant file: `openverifiablellm/utils.py`
- Function: `extract_text_from_xml()`

This issue appears to be logic-related and should be reproducible across environments.

### Impact

High - Major feature is broken

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。