Loading just one particular split is not possible for imagenet-1k

### Describe the bug

I'd expect the following code to download just the validation split but instead I get all data on my disk (train, test and validation splits)

`
from datasets import load_dataset

dataset = load_dataset("imagenet-1k", split="validation", trust_remote_code=True)
`

Is it expected to work like that?

### Steps to reproduce the bug

1. Install the required libraries (python, datasets, huggingface_hub)
2. Login using huggingface cli
2. Run the code in the description

### Expected behavior

Just a single (validation) split should be downloaded.

### Environment info

python: 3.12.2
datasets: 2.18.0
huggingface_hub: 0.22.2