Make sure that our models are usefule for practical use (architecture decisions with inference in mind)

Goal
We want to have a model released that is practically usable.
This means that inference should be reasonably efficient.

Description
With all full attention, our models will be slow and memory heavy. Olmo 3 e.g. used sliding window attention at 3/4 of the layers. We should investigate if the optimal scaling law points show different behavior when switching to such a setting. If nothing changes, potentially, we can switch to such a hybrid for larger scale runs.

Deliverable scope

• run trainings on some scales (of our optimal scaling law points) with sliding window attention on
• test if sliding window attention can be "post-trained" into a existing model (probably not)

Dependencies
none