fsspec / fsspec/filesystem_spec
Failed to read folder
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I noticed a difference in fsspec's handling of folders containing parquet files:
Call method: pd.read_parquet ("s3://xxx/test_dir/")
Normally, if there is a parquet file under the test_dir, this method can read the contents of the parquet file normally. The problem is:
If test_dir is a folder automatically created by aws s3 cp a.parquet s3://xxx/test_dir/, it can be read normally.
But if test_dir a file created by clicking the "Create Folder" button on the s3 console (https://console.amazonaws.cn/), and then uploading a.parquet to this folder, an exception will be thrown: raise FileNotFoundError (path)
The reason is that in line 339 of the fs.py in pyarrow get_file_info_selector () will call this method:
selected_files = self.fs.find (
Selector.base_dir, maxdepth = maxdepth, withdirs = True, detail = True
)
This method calls wrapper () in fsspec's asyn.py and return return sync (self.loop, func, * args, ** kwargs)
The return contains two results: s3://xxx/test_dir/ and s3://xxx/test_dir/a.parquet
After that, pyarrow will read s3://xxx/test_dir/ as a file, which raises the raise FileNotFoundError (path) exception in the fs.py
But the problem is that if the test_dir is a folder automatically created by aws s3 cp a.parquet s3://xxx/test_dir/ the return result of wrapper () does not contain s3://xxx/test_dir/. Therefore, the data can be read normally.
version info:
fsspec 2024.6.1
pyarrow 13.0.0
s3fs 2024.6.1
s3transfer 0.10.2
awscli 1.34.4
aiobotocore 2.13.3
boto3 1.35.4
botocore 1.35.4
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the S3 folder listing behavior described in the issue, then inspect fs.py around get_file_info_selector and asyn.py around wrapper. Compare results for folders created by aws s3 cp and by the S3 console. Done means the console-created folder containing a.parquet can be read through pd.read_parquet without FileNotFoundError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100