Future work on TiME
Open
- Dominant language
- Python
- Stars
- 6
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Hi @dschulmeist,
many thanks for releasing the codebase for the distillation setup! I would like to know if you accidentally working on further research directions of it, as I am super interested to try this approach for ModernBERT architectures.
I think because of RoPE and the altternating local/global attention mechanism it could be tricky, but given the popular widespread of ModernBERT-like models (mmBERT and also Ettin), it could be worth to extend it.
Feel free to share your thoughts on it and many thanks in advance,
Stefan
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.