AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM
[FEATURE]: Add Unit Tests for create_tokenizer and load_merkle_proof
- Ngôn ngữ chính
- Python
- Star
- 18
- Fork
- 31
- Merge trung bình
- 1 phút
- Pull request đã merge (30 ngày)
- 2
Mô tả
### Feature and its Use Cases
### Overview
The repository currently lacks unit tests for two utility functions:
```create_tokenizer``` in ```tests/test_tokenizer.py``` and ```load_merkle_proof``` in ```tests/test_util.py```
* This lack of test coverage creates blind spots, making the code harder to safely refactor and potentially allowing bugs to be introduced without being detected by our test suite.
# Example scenarios to test:
* When "bpe" is passed, the function should return an instance of the BPE tokenizer class.
* When "sentencepiece" is passed, the function should return an instance of the SentencePiece tokenizer class.
* When an unsupported tokenizer type is passed, the function should raise a ValueError.
* Loading a valid JSON Merkle proof file returns the expected structure.
* Attempting to load a non-existent file raises FileNotFoundError.
* Attempting to load invalid JSON raises a parsing exception.
# Acceptance Criteria
* Unit tests exist for both create_tokenizer and load_merkle_proof.
* Tests cover both successful execution and failure scenarios.
* All tests pass when running:
### Additional Context
_No response_
### Code of Conduct
- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.