AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: Complete SentencePiece tokenizer — encode, decode, load and input validation

未关闭
#51 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Python
星标
18
派生
31
平均合并
1 分钟
30 天内合并 PR
2

描述

## Summary

This issue proposes completing the SentencePieceTokenizer introduced in #17. The class currently supports training only. It is missing **encode**(), **decode**(), **load**(), **input validation,** special tokens, and directory creation


## Background

PR #17 introduced a modular tokenizer architecture with SentencePieceTokenizer as one of the two implementations.

However the current SentencePieceTokenizer only covers the training step

A tokenizer that cannot encode or decode text cannot be used in any downstream pipeline task.


## Problems in Current Implementation

### 1. No encode() or decode()

### 2. No load()

### 3. Missing special tokens in training

### 4. No input validation

### 5. save_path directory never created

## Why This Matters

SentencePiece is used by major modern LLMs:

Without a complete SentencePieceTokenizer, the project cannot support cryptographic verification of pipelines
built on any of these models — which represent the majority of current open source LLMs.


## Scope

This is a change touching only:
- sentencepiece_tokenizer.py

No overlap with any existing open PRs.

## Related

- PR #17 — feat: deterministic tokenizer training and config hashing

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。