[Question] Implementing slightly new architecture for slime without invasive updates to native Megatron

Hi,

I'm interested in implementing our custom transformer-based model to slime without writing native megatron if possible. I saw this documentation with an example of Qwen3Next.

We're interested in the following features:

• The model needs custom mask and our model currently uses flex attention in pytorch and flashinfer in SGLang.
• Would context parallelism be possible if we go with this integration?
• You mentioned that TP is not supported "within the replaced module". Could you illustrate a bit more on what defines a replaced module?

Our model is essentially a transformer (e.g. Qwen) with the following differences:

• The input will be preprocessed before sending to the model (i'm sure its doable)
• Different loss calculation (also seems easy to modify)
• Attention pattern is customized per batch
• We hope to apply TP and CP if possible

If TP is not possible for certain modules (attention for example), could you give me some hints on what's the largest possible model size and context length that this approach can support? Really appreciate your help!

Best

Start with docs/en/advanced/arch-support-beyond-megatron.md and its Qwen3Next example to understand the supported integration boundary. Investigate how custom preprocessing, loss, attention, tensor parallelism, and context parallelism map to that approach; done would require a documented feasibility decision or a scoped implementation plan.