Format of Pile and MiniPile for SeqIO

Hi! Thank you for this! Great work.

I've trained with T5X in the past and plugging in my own custom data was the more cumbersome part. So I'm curious, if I follow your scripts in [pile_utils.py](https://github.com/EleutherAI/improved-t5/blob/main/data/pile/pile_utils.py), do I just need to update the bucket in [tasks.py](https://github.com/EleutherAI/improved-t5/blob/main/data/pile/tasks.py) and point it to a folder containing numbered files in which each line is a whole document? Do these documents contain any special token? Is the text tokenized? Are they chunked by sequence length or something?

Sorry for all the questions.