AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[BUG]: hash_tokenizer_config() crashes with FileNotFoundError when used with SentencePiece tokenizer

Abierto
#69 2 comentarios 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Python
Estrellas
18
Forks
31
Merge medio
1 min
PR fusionados (30 d)
2

Descripción


## Summary

`hash_tokenizer_config()` in `openverifiablellm/tokenizer.py` is hardcoded to look for BPE artifacts (`vocab.json` and `merges.txt`). Calling it after training a SentencePiece tokenizer crashes immediately with
`FileNotFoundError` because SentencePiece produces `spm.model` and `spm.vocab` instead.

The function claims to hash tokenizer configuration but silently only works for one of the two supported
tokenizer types.



## Root Cause

In `openverifiablellm/tokenizer.py`, `hash_tokenizer_config()` is hardcoded
for BPE artifacts only

SentencePiece training produces completely different artifacts:
```
spm.model ← binary model file
spm.vocab ← human readable vocab
```

Neither `vocab.json` nor `merges.txt` is ever produced by SentencePiece,
so the function always crashes when called after SentencePiece training.


## Proposed Fix

### `openverifiablellm/tokenizer.py`

Detect tokenizer type from artifacts present on disk

### `tests/test_tokenizer.py`

Add missing tests


## Files to Change

- `openverifiablellm/tokenizer.py` — fix `hash_tokenizer_config()`
- `tests/test_tokenizer.py` — add missing test cases


## Related

- PR #17 — feat: deterministic tokenizer training and config hashing
(introduced `hash_tokenizer_config()` with BPE-only support)
- CodeRabbit review on PR #17 flagged missing `test_hash_changes_when_merges_change`
- Follows from: [FEATURE] Complete SentencePiece tokenizer implementation
- Follows from: [FEATURE] Complete BPETokenizer and BaseTokenizer contract

---

### Impact

Critical - Application is unusable

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.