Differences and exact configs of allenai/OLMo-1B-0724-hf vs allenai/OLMo-1B-hf
Aperta
type/question
- Lingua principale
- Python
- Stelle
- 6.7k
- Fork
- 797
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
### ❓ The question
Hi! I'm trying to understand the differences between these two models. Looking at the HF configs and model pages I found the following ones:
- Difference training process, 1 stage vs 2 stages and different version of dolma v1_5 vs v_1_7
- Context length for training, 2048 vs 4096
- embedding weight tying , tied vs untied
- clip_qkv, null vs 8.0
I also looked at the config at `configs/official-0724/OLMo-1B.yaml` but can't figure out it is supposed to be the 0724 version or the first one. Are there training runs/exact configs for these exact models?
Thanks for any info!
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.