allenai / allenai/OLMo

Differences and exact configs of allenai/OLMo-1B-0724-hf vs allenai/OLMo-1B-hf

Aperta
#901 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
type/question
Lingua principale
Python
Stelle
6.7k
Fork
797
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### ❓ The question

Hi! I'm trying to understand the differences between these two models. Looking at the HF configs and model pages I found the following ones:

- Difference training process, 1 stage vs 2 stages and different version of dolma v1_5 vs v_1_7
- Context length for training, 2048 vs 4096
- embedding weight tying , tied vs untied
- clip_qkv, null vs 8.0

I also looked at the config at `configs/official-0724/OLMo-1B.yaml` but can't figure out it is supposed to be the 0724 version or the first one. Are there training runs/exact configs for these exact models?

Thanks for any info!

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.