[Python] Creating dataset with an S3 Path using fsspec filesystem does not work
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
I was testing loading datasets using fsspec filesystem instead of the arrow ones, just for the sake of testing it out. What I found is:
Given a S3 server, working keys and a dataset_path that exists and works OK.
If I try to create a dataset based on a dataset_path (directory) as a relative path from the root directory
Let us use a fake value in the code as an example (cannot share original path) so we can see the issue easier:
```
import pyarrow.dataset as pd
import fsspec
dataset_path = "fake_directory/fake_dataset"
app_keys = {
'client_kwargs':{"endpoint_url": s3_host},
'key': access_key,
'secret': secret_key,
}
s3_fs = fsspec.filesystem("s3", **app_keys)
dataset = pd.dataset(dataset_path, filesystem=s3_fs, format="parquet", partitioning="hive")
#Traceback (most recent call last):
# ...
# dataset = pd.dataset(f"{dataset_path}", filesystem=s3_fs, format="parquet", partitioning="hive")
# File "C:\Repos\project\venv\lib\site-packages\pyarrow\dataset.py", line 752, in dataset
# return _filesystem_dataset(source, **kwargs)
# File "C:\Repos\project\venv\lib\site-packages\pyarrow\dataset.py", line 454, in _filesystem_dataset
# return factory.finish(schema)
# File "pyarrow\_dataset.pyx", line 1940, in pyarrow._dataset.DatasetFactory.finish
# File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
# File "pyarrow\_fs.pyx", line 1551, in pyarrow._fs._cb_open_input_file
# File "C:\Repos\project\venv\lib\site-packages\pyarrow\fs.py", line 419, in open_input_file
# raise FileNotFoundError(path)
#FileNotFoundError: fake_directory/fake_dataset
```
If the dataset_path is given as absolute (s3://{dataset_path} or simply /{dataset_path}), the same code throws this error:
```
import pyarrow.dataset as pd
import fsspec
dataset_path = "/fake_directory/fake_dataset"
app_keys = {
'client_kwargs':{"endpoint_url": s3_host},
'key': access_key,
'secret': secret_key,
}
s3_fs = fsspec.filesystem("s3", **app_keys)
dataset = pd.dataset(dataset_path, filesystem=s3_fs, format="parquet", partitioning="hive")
#Traceback (most recent call last):
# ...
# dataset = pd.dataset(f"/{dataset_path}", filesystem=s3_fs, format="parquet", partitioning="hive")
# File "C:\Repos\project\venv\lib\site-packages\pyarrow\dataset.py", line 752, in dataset
# return _filesystem_dataset(source, **kwargs)
# File "C:\Repos\project\venv\lib\site-packages\pyarrow\dataset.py", line 452, in _filesystem_dataset
# factory = FileSystemDatasetFactory(fs, paths_or_selector, format, options)
# File "pyarrow\_dataset.pyx", line 2114, in pyarrow._dataset.FileSystemDatasetFactory.__init__
# File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
# File "pyarrow\error.pxi", line 100, in pyarrow.lib.check_status
#pyarrow.lib.ArrowInvalid: GetFileInfo() yielded path 'fake_directory/fake_dataset/{rest_of_path_to_a_file}', which is outside base dir '/fake_directory/fake_dataset'
```
First thought was this just may not be an intended feature. However, if instead of a directory, we pass a file or a list of files, the dataset is created just fine. So seeing this is supported led me to open this issue.
Also, the dataset is working fine using dataset_path using SubTreeFileSystem, so it is not any kind of access problem, and the fsspec filesystem object created is able to list files just fine.
To me, it looks like some kind of issue in the interaction between fsspec and arrow when discovering the dataset composition, which leads to confuse a directory to a file.
Using Python 3.10.0, fsspec 2022.11.0 and tested in on both pyarrow 9.0.0 and 10.0.1
### Component(s)
Python
Contributor guide
Research direction
Reproduce the directory case with pyarrow.dataset, fsspec, and the S3 filesystem setup described in the report, then compare it with the working file and list-of-files cases. Start at _filesystem_dataset, FileSystemDatasetFactory, and open_input_file; done means a relative or absolute directory path works with the fsspec filesystem while preserving the existing successful cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100