AOSSIE-Org / AOSSIE-Org/OpenVerifiableLLM
[FEATURE]: Implement Deterministic Dataset Encoding Pipeline and Verifiable LLaMA Model (Model Architecture)
- Linguagem predominante
- Python
- Estrelas
- 18
- Forks
- 31
- Merge médio
- 1min
- PRs com merge (30d)
- 2
Descrição
# deterministic dataset encoding pipeline and a minimal implementation of a LLaMA architecture.
* The dataset is currently processed into wiki_clean.txt and a tokenizer has been trained. We need to implement a memory-efficient script to encode the entire dataset into binary format for training.
* Implement a minimal PyTorch LLaMA-style architecture to ensure deterministic behavior and full control over initialization.
* Read the dataset in chunks to avoid high memory usage.
* Use the trained tokenizer (BPE/SentencePiece) to convert text into token IDs.
* Stream token IDs into a binary dataset file (.bin, uint16 or similar).
* Compute a SHA256 hash of the resulting file.
# Verification Criteria
* Running the dataset encoding pipeline twice should produce identical binary files and SHA256 hashes.
* Initializing the model twice with the same seed should produce identical parameter hashes.
must output the exact same initial parameter hashes.
### Additional Context
_No response_
### Code of Conduct
- [x] I have joined the [Discord server](https://discord.gg/hjUhu33uAn) and will post updates there
- [x] I have searched existing issues to avoid duplicates
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.