AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: Add Unit Tests for create_tokenizer and load_merkle_proof

未关闭
#77 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Python
星标
18
派生
31
平均合并
1 分钟
30 天内合并 PR
2

描述

### Feature and its Use Cases

### Overview
The repository currently lacks unit tests for two utility functions:
```create_tokenizer``` in ```tests/test_tokenizer.py``` and ```load_merkle_proof``` in ```tests/test_util.py```
* This lack of test coverage creates blind spots, making the code harder to safely refactor and potentially allowing bugs to be introduced without being detected by our test suite.

# Example scenarios to test:
* When "bpe" is passed, the function should return an instance of the BPE tokenizer class.
* When "sentencepiece" is passed, the function should return an instance of the SentencePiece tokenizer class.
* When an unsupported tokenizer type is passed, the function should raise a ValueError.
* Loading a valid JSON Merkle proof file returns the expected structure.
* Attempting to load a non-existent file raises FileNotFoundError.
* Attempting to load invalid JSON raises a parsing exception.

# Acceptance Criteria
* Unit tests exist for both create_tokenizer and load_merkle_proof.
* Tests cover both successful execution and failure scenarios.
* All tests pass when running:

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。