s3fs throws FileNotFound Exception while reading updated file through fastparquet api
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
Hi,
I am using s3fs to read/write s3 files with fastparquet library. fastparquet provides python interface to read/write parquet format files. Please refer below block to understand how s3fs is used with fastparquet library to read/write parquet files.
import s3fs
from fastparquet import ParquetFile
s3 = s3fs.S3FileSystem()
myopen = s3.open
pf = ParquetFile('/mybucket/data.parquet', open_with=myopen)
df = pf.to_pandas()
The function myopen provided to the constructor must be callable with f(path, mode) and produce an open file context.
If file is updated/modified then using same s3fs connection object throws FileNotFound Exception.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/fastparquet/api.py", line 110,
in __init__ with open_with(fn2, \'rb\') as f:
File "/usr/local/lib/python3.6/dist-packages/fsspec/spec.py", line 689, in open
autocommit=ac, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/s3fs/core.py", line 314, in _open autocommit=autocommit)
File "/usr/local/lib/python3.6/dist-packages/s3fs/core.py", line 939, in __init__ cache_type=cache_type)
File "/usr/local/lib/python3.6/dist-packages/fsspec/spec.py", line 884, in __init__
self.details = fs.info(path)
File "/usr/local/lib/python3.6/dist-packages/s3fs/core.py", line 498, in info
return super().info(path)
File "/usr/local/lib/python3.6/dist-packages/fsspec/spec.py", line 509, in info
raise FileNotFoundError(path)\nFileNotFoundError: mybucket/data.parquet/_metadata
During investigation of this issue, I found that S3FileSystem class inherits python's AbstractFileSystem class. AbstractFileSystem (An abstract super-class for pythonic file-systems) inherits _Cached (Metaclass for caching file system instances) class.
It fails to detect s3 file modifications as it has already cached file instance for that file.
Please explain this issue. Whether it is design flow or extra params to be passed to enable S3FileSystem class to detect file level modifications.
To use the same s3fs connection, you can disable file instance caching as below.
import s3fs
from fastparquet import ParquetFile
s3fs.S3FileSystem.cachable = False
s3 = s3fs.S3FileSystem()
myopen = s3.open
pf = ParquetFile('/mybucket/data.parquet', open_with=myopen)
df = pf.to_pandas()
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the fastparquet read of an updated S3 object with one S3FileSystem connection, following the traceback through fsspec/spec.py and s3fs/core.py. Check the S3FileSystem instance and file caching behavior described in the issue, then document whether the current behavior is intentional or identify the required change and a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100