fsspec / fsspec/s3fs

s3fs throws FileNotFound Exception while reading updated file through fastparquet api

Open
#272 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

Hi,
I am using s3fs to read/write s3 files with fastparquet library. fastparquet provides python interface to read/write parquet format files. Please refer below block to understand how s3fs is used with fastparquet library to read/write parquet files.

import s3fs      
from fastparquet import ParquetFile
s3 = s3fs.S3FileSystem()
myopen = s3.open
pf = ParquetFile('/mybucket/data.parquet', open_with=myopen)
df = pf.to_pandas()

The function myopen provided to the constructor must be callable with f(path, mode) and produce an open file context.

If file is updated/modified then using same s3fs connection object throws FileNotFound Exception.

Traceback (most recent call last):  
File "/usr/local/lib/python3.6/dist-packages/fastparquet/api.py", line 110,
in __init__    with open_with(fn2, \'rb\') as f:
File "/usr/local/lib/python3.6/dist-packages/fsspec/spec.py", line 689, in open    
autocommit=ac, **kwargs)  
File "/usr/local/lib/python3.6/dist-packages/s3fs/core.py", line 314, in _open    autocommit=autocommit)
File "/usr/local/lib/python3.6/dist-packages/s3fs/core.py", line 939, in __init__    cache_type=cache_type)  
File "/usr/local/lib/python3.6/dist-packages/fsspec/spec.py", line 884, in __init__    
self.details = fs.info(path)  
File "/usr/local/lib/python3.6/dist-packages/s3fs/core.py", line 498, in info   
return super().info(path)  
File "/usr/local/lib/python3.6/dist-packages/fsspec/spec.py", line 509, in info
raise FileNotFoundError(path)\nFileNotFoundError: mybucket/data.parquet/_metadata

During investigation of this issue, I found that S3FileSystem class inherits python's AbstractFileSystem class. AbstractFileSystem (An abstract super-class for pythonic file-systems) inherits _Cached (Metaclass for caching file system instances) class.
It fails to detect s3 file modifications as it has already cached file instance for that file.
Please explain this issue. Whether it is design flow or extra params to be passed to enable S3FileSystem class to detect file level modifications.

To use the same s3fs connection, you can disable file instance caching as below.

import s3fs      
from fastparquet import ParquetFile
s3fs.S3FileSystem.cachable = False
s3 = s3fs.S3FileSystem()
myopen = s3.open
pf = ParquetFile('/mybucket/data.parquet', open_with=myopen)
df = pf.to_pandas()

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the fastparquet read of an updated S3 object with one S3FileSystem connection, following the traceback through fsspec/spec.py and s3fs/core.py. Check the S3FileSystem instance and file caching behavior described in the issue, then document whether the current behavior is intentional or identify the required change and a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.