Agree with pretraining team on the context extension parameters

Highlevel, this task comes to optimizing the full pipeline on the models we want to deliver and the expectations on those models.
Between the pre- and post-training, we need to agree on optimization of the long context capability.

Key questions to anwer:
1. what is the amount of compute / data we should use for the context extension?

The amount of compute is decisive factor on deciding what options we have for context extension.

We can roughly classify the ways to do context extension as follows:

• Pretraining type of training: Continue with pretraining objective, but use longer sequence length data and training than during pretraining. Also some synthetic data may be added to the training dataset here
• RoPE (Rotary position embedding) scaling related training: We use longer sequence length in addition to RoPE scaling to teach the model long context relationships. This does not require that long training, but in practice often does not work as well as 1.
• Hybrid/combination of the above, where type 1 training is first used and it is further extended with type 2 training.

There are also some architecture related matters that matter here as the attention calculation operation is exponential in compute relative to sequence length, so the longer sequence lengths are always more expensive. Some new architectural choices like use of linear attention matter and they may be negligible in cost on pretraining but matter in context extension / post training. Also long context performance may be optimal with different kind of model than with shorter sequence model performance. One new paper regarding this https://allenai.org/papers/olmpool

2. which parameters are used in pretraining, and what is adequate for long context purposes?

There are other parameters like RoPE base frequency (AKA RoPE Theta) that needs some discussion. Pretraining has done most of the experiments using RoPE base of 10000 which is quite small for long context purposes and this should be probably changed in some part of training.