AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: Add Unit Tests for create_tokenizer and load_merkle_proof

Đang mở
#77 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
Python
Star
18
Fork
31
Merge trung bình
1 phút
Pull request đã merge (30 ngày)
2

Mô tả

### Feature and its Use Cases

### Overview
The repository currently lacks unit tests for two utility functions:
```create_tokenizer``` in ```tests/test_tokenizer.py``` and ```load_merkle_proof``` in ```tests/test_util.py```
* This lack of test coverage creates blind spots, making the code harder to safely refactor and potentially allowing bugs to be introduced without being detected by our test suite.

# Example scenarios to test:
* When "bpe" is passed, the function should return an instance of the BPE tokenizer class.
* When "sentencepiece" is passed, the function should return an instance of the SentencePiece tokenizer class.
* When an unsupported tokenizer type is passed, the function should raise a ValueError.
* Loading a valid JSON Merkle proof file returns the expected structure.
* Attempting to load a non-existent file raises FileNotFoundError.
* Attempting to load invalid JSON raises a parsing exception.

# Acceptance Criteria
* Unit tests exist for both create_tokenizer and load_merkle_proof.
* Tests cover both successful execution and failure scenarios.
* All tests pass when running:

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.