Bias Term in Router
Ouverte
- Langage dominant
- Python
- Étoiles
- 155
- Forks
- 24
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
Hi Weijia and Akshita,
Awesome work! I noticed in the paper that there is a negative bias term for the router, but the released model [allenai/FlexOlmo-7x7B-1T](https://huggingface.co/allenai/FlexOlmo-7x7B-1T) and the huggingface [transformers code](https://github.com/huggingface/transformers/blob/main/src/transformers/models/flex_olmo/modeling_flex_olmo.py#L336) don't have a negative bias.
Did you guys happen to use the negative router bias for the final released model?
Thanks!
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.