AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: Add Unit Tests for create_tokenizer and load_merkle_proof

Abierto
#77 1 comentario 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
18
Forks
31
Merge medio
1 min
PR fusionados (30 d)
2

Descripción

### Feature and its Use Cases

### Overview
The repository currently lacks unit tests for two utility functions:
```create_tokenizer``` in ```tests/test_tokenizer.py``` and ```load_merkle_proof``` in ```tests/test_util.py```
* This lack of test coverage creates blind spots, making the code harder to safely refactor and potentially allowing bugs to be introduced without being detected by our test suite.

# Example scenarios to test:
* When "bpe" is passed, the function should return an instance of the BPE tokenizer class.
* When "sentencepiece" is passed, the function should return an instance of the SentencePiece tokenizer class.
* When an unsupported tokenizer type is passed, the function should raise a ValueError.
* Loading a valid JSON Merkle proof file returns the expected structure.
* Attempting to load a non-existent file raises FileNotFoundError.
* Attempting to load invalid JSON raises a parsing exception.

# Acceptance Criteria
* Unit tests exist for both create_tokenizer and load_merkle_proof.
* Tests cover both successful execution and failure scenarios.
* All tests pass when running:

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.