Unbearably slow sorting on big mapped datasets

### Describe the bug

For me, with ~40k lines, sorting took 3.5 seconds on a flattened dataset (including the flatten operation) and 22.7 seconds on a mapped dataset (right after sharding), which is about x5 slowdown. Moreover, it seems like it slows down exponentially with bigger datasets (wasn't able to sort 700k lines at all, with flattening takes about a minute).

### Steps to reproduce the bug

```Python
from datasets import load_dataset
import time

dataset = load_dataset("xnli", "en", split="train")

dataset = dataset.shard(10, 0)

print(len(dataset))

t = time.time()

# dataset = dataset.flatten_indices() # uncomment this line and it's fast

dataset = dataset.sort("label", reverse=True, load_from_cache_file=False)

print(f"finished in {time.time() - t:.4f} seconds")

```

### Expected behavior

Expect sorting to take the same or less time than flattening and then sorting.

### Environment info

- `datasets` version: 2.12.1.dev0 (same with 2.12.0 too)
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1