huggingface / huggingface/datasets
Tilde (~) is not supported for data_files
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
Attempting to `load_dataset` from a path starting with `~` (as a shorthand for the user's home directory) seems not to be fully working - at least as far as the `parquet` dataset builder is concerned.
(the same file can be loaded correctly if providing its absolute path instead)
I think that this is very similar to https://github.com/huggingface/datasets/issues/5757, but for `data_files` rather than `data_dir`
### Steps to reproduce the bug
```python
import datasets
# save a parquet file at ~/path/to/data.parquet
data_files = "~/path/to/data.parquet"
dataset = datasets.load_dataset("parquet", data_files=data_files)
```
```
Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 12671.61it/s]
Extracting data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 22671.91it/s]
Generating train split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1949, in _prepare_split_single
num_examples, num_bytes = writer.finalize()
^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/datasets/arrow_writer.py", line 598, in finalize
raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File ".venv/lib/python3.11/site-packages/datasets/load.py", line 2133, in load_dataset
builder_instance.download_and_prepare(
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 954, in download_and_prepare
self._download_and_prepare(
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1813, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1958, in _prepare_split_single
raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```
### Expected behavior
Can use `~` shorthand in paths when loading local (parquet) datasets.
### Environment info
`datasets 2.14.3`
Contributor guide
Assessment
This issue has not been assessed yet.