AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[BUG]: hash_tokenizer_config() crashes with FileNotFoundError when used with SentencePiece tokenizer

Đang mở
#69 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
bug
Ngôn ngữ chính
Python
Star
18
Fork
31
Merge trung bình
1 phút
Pull request đã merge (30 ngày)
2

Mô tả


## Summary

`hash_tokenizer_config()` in `openverifiablellm/tokenizer.py` is hardcoded to look for BPE artifacts (`vocab.json` and `merges.txt`). Calling it after training a SentencePiece tokenizer crashes immediately with
`FileNotFoundError` because SentencePiece produces `spm.model` and `spm.vocab` instead.

The function claims to hash tokenizer configuration but silently only works for one of the two supported
tokenizer types.



## Root Cause

In `openverifiablellm/tokenizer.py`, `hash_tokenizer_config()` is hardcoded
for BPE artifacts only

SentencePiece training produces completely different artifacts:
```
spm.model ← binary model file
spm.vocab ← human readable vocab
```

Neither `vocab.json` nor `merges.txt` is ever produced by SentencePiece,
so the function always crashes when called after SentencePiece training.


## Proposed Fix

### `openverifiablellm/tokenizer.py`

Detect tokenizer type from artifacts present on disk

### `tests/test_tokenizer.py`

Add missing tests


## Files to Change

- `openverifiablellm/tokenizer.py` — fix `hash_tokenizer_config()`
- `tests/test_tokenizer.py` — add missing test cases


## Related

- PR #17 — feat: deterministic tokenizer training and config hashing
(introduced `hash_tokenizer_config()` with BPE-only support)
- CodeRabbit review on PR #17 flagged missing `test_hash_changes_when_merges_change`
- Follows from: [FEATURE] Complete SentencePiece tokenizer implementation
- Follows from: [FEATURE] Complete BPETokenizer and BaseTokenizer contract

---

### Impact

Critical - Application is unusable

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.