Question about masking - why mask on unpadded sequence?

Hi @warner-benjamin @ohmeow,
Nice work on the ModernBERT project — I’ve been learning a lot from it.

I have one question regarding the masking logic here:
https://github.com/AnswerDotAI/ModernBERT/blob/8c57a0f01c12c4953ead53d398a36f81a4ba9e38/src/sequence_packer.py#L264-L281

From what I understand, masking is applied **after** the sequence packing step.
This means that the masking probability is applied across the entire packed sequence (pseq), without regard to the original sample boundaries. As a result, some original samples inside a packed sequence might end up with no masked tokens at all.

I was curious about the intent behind applying masking at the packed-sequence level rather than per original sample.
Could you share the reasoning or trade-offs for this design choice?

Thanks,
- Youngjoon Jang

Read src/sequence_packer.py at lines 264-281 and compare the masking step with the sequence-packing behavior described in the issue. Trace how masking relates to packed-sequence and original-sample boundaries; done means documenting the intent and trade-offs in a maintainer-approved response or project documentation.

Written by the indexing model from the issue text.