[QUESTION] Dataset construction behavior for variable-length documents

**Your question**
I'd like to know the dataset preprocessing behavior when working with documents of varying lengths. Here's my scenario:

- I've preprocessed a dataset containing 128k-token sequence
- I'm training with seq_length=32k

My questions regarding the dataset construction process:

For documents exceeding 32k tokens (like my 128k-token samples), does the `build_sample_idx` (https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/datasets/helpers.cpp#L143) function truncate them to the sequence length?

For documents shorter than 32k tokens, does the implementation perform sequence packing to combine multiple short documents into ~32k-length training samples?

Could you please confirm this behavior? I want to ensure I understand how the dataset builder handles both long and short documents relative to the target sequence length.

Thank you for your reply!