AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM
[FEATURE]: Implement Deterministic Dataset Encoding Pipeline and Verifiable LLaMA Model (Model Architecture)
- 主要言語
- Python
- スター
- 18
- フォーク
- 31
- 平均マージ
- 1分
- マージ済み PR(30日)
- 2
説明
# deterministic dataset encoding pipeline and a minimal implementation of a LLaMA architecture.
* The dataset is currently processed into wiki_clean.txt and a tokenizer has been trained. We need to implement a memory-efficient script to encode the entire dataset into binary format for training.
* Implement a minimal PyTorch LLaMA-style architecture to ensure deterministic behavior and full control over initialization.
* Read the dataset in chunks to avoid high memory usage.
* Use the trained tokenizer (BPE/SentencePiece) to convert text into token IDs.
* Stream token IDs into a binary dataset file (.bin, uint16 or similar).
* Compute a SHA256 hash of the resulting file.
# Verification Criteria
* Running the dataset encoding pipeline twice should produce identical binary files and SHA256 hashes.
* Initializing the model twice with the same seed should produce identical parameter hashes.
must output the exact same initial parameter hashes.
### Additional Context
_No response_
### Code of Conduct
- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates
コントリビューションガイド
評価
この issue はまだ評価されていません。