AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM

[FEATURE]: Implement Deterministic Dataset Encoding Pipeline and Verifiable LLaMA Model (Model Architecture)

オープン
#55 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
enhancement
主要言語
Python
スター
18
フォーク
31
平均マージ
1分
マージ済み PR(30日)
2

説明

# deterministic dataset encoding pipeline and a minimal implementation of a LLaMA architecture.

* The dataset is currently processed into wiki_clean.txt and a tokenizer has been trained. We need to implement a memory-efficient script to encode the entire dataset into binary format for training.

* Implement a minimal PyTorch LLaMA-style architecture to ensure deterministic behavior and full control over initialization.
* Read the dataset in chunks to avoid high memory usage.

* Use the trained tokenizer (BPE/SentencePiece) to convert text into token IDs.

* Stream token IDs into a binary dataset file (.bin, uint16 or similar).

* Compute a SHA256 hash of the resulting file.

# Verification Criteria

* Running the dataset encoding pipeline twice should produce identical binary files and SHA256 hashes.

* Initializing the model twice with the same seed should produce identical parameter hashes.

must output the exact same initial parameter hashes.

### Additional Context

_No response_

### Code of Conduct

- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。