Sample packing for pretraining/fine-tuning

Nobody has claimed this yet.

I was wondering if there are sample packing approaches defined somewhere for preprocessing and tokenization of datasets? I looked through different prepare_*.py, but couldn't find anything related to packing multiple sequences being packed into max_length for efficiency etc

Also, wondering how the data prep works as of now in the lit-gpt framework:

- if a document, article, instruction/output pair exceeds the max sequence length, how is it treated? What about if a doc/article/instruction-output pair falls short of max seq. length? are the remaining time steps padded or are more sequences packed until max length is achieved?

Start by reviewing the prepare_*.py scripts and the existing preprocessing and tokenization flow in lit-gpt. Determine the current handling of sequences longer or shorter than max_length, then define the desired sample-packing behavior and its completion criteria; the issue does not identify a target file or test.

Written by the indexing model from the issue text.