[Question] Usage of sep token in prepare_redpajama.py

Nobody has claimed this yet.

Hi,

While reviewing the [`prepare_redpajama.py`](https://github.com/Lightning-AI/lit-gpt/blob/main/scripts/prepare_redpajama.py) script, I came across the preprocessing step involving the `PackedDatasetBuilder` class. I noticed that during initialization, the class seems to fill the `_arr` entirely with `_sep_token`.

However, it caught my attention that the `add_array` method does not insert `_sep_token` between different documents. It appears that the `_sep_token` is only present at the end of the last chunk. I am curious if this is intentional, possibly because there is no need to mark the end of documents during pretraining procedure?

Thank you for your time and assistance.

cc @awaelchli

Start with scripts/prepare_redpajama.py and trace the PackedDatasetBuilder initialization, _arr contents, and add_array method. Compare how _sep_token is handled across document boundaries and chunks. Done means the intended behavior is established and documented clearly, or a focused follow-up change is identified.

Written by the indexing model from the issue text.