AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM
[BUG]: hash_tokenizer_config() crashes with FileNotFoundError when used with SentencePiece tokenizer
- Lenguaje dominante
- Python
- Estrellas
- 18
- Forks
- 31
- Merge medio
- 1 min
- PR fusionados (30 d)
- 2
Descripción
## Summary
`hash_tokenizer_config()` in `openverifiablellm/tokenizer.py` is hardcoded to look for BPE artifacts (`vocab.json` and `merges.txt`). Calling it after training a SentencePiece tokenizer crashes immediately with
`FileNotFoundError` because SentencePiece produces `spm.model` and `spm.vocab` instead.
The function claims to hash tokenizer configuration but silently only works for one of the two supported
tokenizer types.
## Root Cause
In `openverifiablellm/tokenizer.py`, `hash_tokenizer_config()` is hardcoded
for BPE artifacts only
SentencePiece training produces completely different artifacts:
```
spm.model ← binary model file
spm.vocab ← human readable vocab
```
Neither `vocab.json` nor `merges.txt` is ever produced by SentencePiece,
so the function always crashes when called after SentencePiece training.
## Proposed Fix
### `openverifiablellm/tokenizer.py`
Detect tokenizer type from artifacts present on disk
### `tests/test_tokenizer.py`
Add missing tests
## Files to Change
- `openverifiablellm/tokenizer.py` — fix `hash_tokenizer_config()`
- `tests/test_tokenizer.py` — add missing test cases
## Related
- PR #17 — feat: deterministic tokenizer training and config hashing
(introduced `hash_tokenizer_config()` with BPE-only support)
- CodeRabbit review on PR #17 flagged missing `test_hash_changes_when_merges_change`
- Follows from: [FEATURE] Complete SentencePiece tokenizer implementation
- Follows from: [FEATURE] Complete BPETokenizer and BaseTokenizer contract
---
### Impact
Critical - Application is unusable
### Code of Conduct
- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.