AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: Complete BPETokenizer and BaseTokenizer-Add encode, decode and load to BaseTokenizer and BPETokenizer

Đang mở
#52 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
Python
Star
18
Fork
31
Merge trung bình
1 phút
Pull request đã merge (30 ngày)
2

Mô tả


## Summary

This issue proposes adding encode(), decode(), and load() to BaseTokenizer as abstract methods and implementing them fully in BPETokenizer. This completes the BPE tokenizer
contract and ensures all future tokenizer implementations
are required to support these core operations.


## Background

PR #17 introduced `BaseTokenizer` as an abstract interface and `BPETokenizer` as its first implementation.

The current `BaseTokenizer` defines:
```
train() ✅ defined
get_vocab_path() ✅ defined
get_merges_path() ✅ defined

encode() ❌ not defined
decode() ❌ not defined
load() ❌ not defined
```

Since encode(), decode() and load() are not part of the abstract interface, subclasses have no obligation to
implement them. This breaks the tokenizer contract and leaves BPETokenizer only half complete.

---

## Problems in Current Implementation

### 1. BaseTokenizer does not enforce encode/decode/load

### 2. BPETokenizer cannot encode or decode

### 3. No way to reload BPETokenizer from disk

### 4. No input validation in BPETokenizer.train()

---

## Proposed Changes

### `openverifiablellm/tokenizer/base.py`

**Add three abstract methods:**

### `openverifiablellm/tokenizer/bpe_tokenizer.py`

**Implement encode():**

**Implement decode():**

**Implement load():**

**Harden train():**

### `tests/test_bpe.py` ← new file

With this PR:

- BaseTokenizer enforces full workflow
- BPETokenizer is fully usable
- encode → decode works
- Pipeline can progress to model training

---


## Scope

This is a change touching only:
- `base.py`
- `bpe_tokenizer.py`
- `tests/test_bpe.py`

No overlap with any existing open PRs.
No dependency on any other issue.

---

## Related

- PR #17 — feat: deterministic tokenizer training and config hashing

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.