huggingface / huggingface/datasets

Tilde (~) is not supported for data_files

Open
#6,212 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

Attempting to `load_dataset` from a path starting with `~` (as a shorthand for the user's home directory) seems not to be fully working - at least as far as the `parquet` dataset builder is concerned.

(the same file can be loaded correctly if providing its absolute path instead)

I think that this is very similar to https://github.com/huggingface/datasets/issues/5757, but for `data_files` rather than `data_dir`

### Steps to reproduce the bug

```python
import datasets

# save a parquet file at ~/path/to/data.parquet

data_files = "~/path/to/data.parquet"
dataset = datasets.load_dataset("parquet", data_files=data_files)
```
```
Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 12671.61it/s]
Extracting data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 22671.91it/s]
Generating train split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1949, in _prepare_split_single
num_examples, num_bytes = writer.finalize()
^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/datasets/arrow_writer.py", line 598, in finalize
raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File ".venv/lib/python3.11/site-packages/datasets/load.py", line 2133, in load_dataset
builder_instance.download_and_prepare(
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 954, in download_and_prepare
self._download_and_prepare(
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1813, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1958, in _prepare_split_single
raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

```

### Expected behavior

Can use `~` shorthand in paths when loading local (parquet) datasets.

### Environment info

`datasets 2.14.3`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.