Unclear how to preprocess entropy data

## How to preprocess data for training?

I am trying to launch the debug script after installation,
```
python -m bytelatent.train config=bytelatent/configs/debug.yaml
```
and I see that I need to specify `preprocess_dir` in the yaml file for training. However, there is no instruction for how to preprocess data. (I guess we should use `preprocess_entropies.py` or `parallel_entropies.py`.)

### What I tried

I attempted to preprocess `.jsonl` files (in `fineweb_edu_10bt_shuffled` dataset) one by one using:

```bash
python -m bytelatent.preprocess.preprocess_entropies fineweb_edu_10bt.chunk.00.jsonl output_dir
```

However, I am unsure about which .pth file should be specified for the entropy model:
- consolidated_with_rope.pth
- consolidated.pth

For enhancing reproducibility, could you please provide a simple guide or script for preprocessing data for training? Many thanks!

Start with bytelatent/configs/debug.yaml and compare the mentioned preprocess_entropies.py and parallel_entropies.py entry points, including the two listed entropy-model checkpoints. Done means a reproducible guide or script explains the preprocessing command, output location for preprocess_dir, and which checkpoint to use for the debug training run.

Written by the indexing model from the issue text.