Modifying step 1 supervised fine-tuning for non-chat datasets?

This is a pretty silly question, but I wanted to confirm if the step 1 SFT scripts such as this one can be modified to fine-tune models for non-chat datasets. So, if I were to use inputs without the Human: Assistant: format, and just have inputs which are paragraphs of text, is that fine?

From what I can see, I would just need to create a custom PromptRawDataset class. I would then make get_prompt_and_chosen() load the text, and also set --data_split 10,0,0. Is this correct?

For my use case, I am basically trying to reproduce some of the experiments of the BLOOMZ paper, at a very small scale. I'm starting from bigscience/bloomz-3b, and fine-tuning on some multilingual tasks. If I can use the deepspeed examples here, it would be great.

Start with the linked step 1 script, then inspect the PromptRawDataset interface and the --data_split handling mentioned in the issue. Determine whether paragraph-only multilingual data can use the existing SFT flow or requires a custom dataset class; done means the required changes and limitations are documented.