`with_format` behavior is inconsistent on different datasets

## Describe the bug
I found a case where `with_format` does not transform the dataset to the requested format.

## Steps to reproduce the bug
Run:

```python
from transformers import AutoTokenizer, AutoFeatureExtractor
from datasets import load_dataset

raw = load_dataset("glue", "sst2", split="train")
raw = raw.select(range(100))

tokenizer = AutoTokenizer.from_pretrained("philschmid/tiny-bert-sst2-distilled")

def preprocess_func(examples):
return tokenizer(examples["sentence"], padding=True, max_length=256, truncation=True)

data = raw.map(preprocess_func, batched=True)

print(type(data[0]["input_ids"]))

data = data.with_format("torch", columns=["input_ids"])

print(type(data[0]["input_ids"]))
```

printing as expected:

```python

```

Then run:

```python
raw = load_dataset("beans", split="train")
raw = raw.select(range(100))

preprocessor = AutoFeatureExtractor.from_pretrained("nateraw/vit-base-beans")

def preprocess_func(examples):
imgs = [img.convert("RGB") for img in examples["image"]]
return preprocessor(imgs)

data = raw.map(preprocess_func, batched=True)

print(type(data[0]["pixel_values"]))

data = data.with_format("torch", columns=["pixel_values"])

print(type(data[0]["pixel_values"]))
```

Printing, unexpectedly

```python

```

## Expected results
`with_format` should transform into the requested format; it's not the case.

## Actual results
`type(data[0]["pixel_values"])` should be `torch.Tensor` in the example above

## Environment info

- `datasets` version: dev version, commit 44af3fafb527302282f6b6507b952de7435f0979
- Platform: Linux
- Python version: 3.9.12
- PyArrow version: 7.0.0