AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: Add Unit Tests for create_tokenizer and load_merkle_proof

Ouverte
#77 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
Python
Étoiles
18
Forks
31
Merge moyen
1 min
PR mergées (30 j)
2

Description

### Feature and its Use Cases

### Overview
The repository currently lacks unit tests for two utility functions:
```create_tokenizer``` in ```tests/test_tokenizer.py``` and ```load_merkle_proof``` in ```tests/test_util.py```
* This lack of test coverage creates blind spots, making the code harder to safely refactor and potentially allowing bugs to be introduced without being detected by our test suite.

# Example scenarios to test:
* When "bpe" is passed, the function should return an instance of the BPE tokenizer class.
* When "sentencepiece" is passed, the function should return an instance of the SentencePiece tokenizer class.
* When an unsupported tokenizer type is passed, the function should raise a ValueError.
* Loading a valid JSON Merkle proof file returns the expected structure.
* Attempting to load a non-existent file raises FileNotFoundError.
* Attempting to load invalid JSON raises a parsing exception.

# Acceptance Criteria
* Unit tests exist for both create_tokenizer and load_merkle_proof.
* Tests cover both successful execution and failure scenarios.
* All tests pass when running:

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.