fsspec / fsspec/s3fs

S3 list operation requested when file not found

Open
#382 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

What happened:
When trying to open a file that does not exist in an AWS S3 bucket, a list operation is requested. In addition, a FileNotFoundError exception is raised.

What you expected to happen:
I would expect to see a FileNotFoundError exception being raised only. No list operations should be requested. It seems unnecessary and AWS S3 charges for it.

Minimal Complete Verifiable Example:
When running:

import s3fs


storage_options = {
    'key': 'my-key',
    'secret': 'my-secret',
    'use_ssl': False,
    'client_kwargs': {
        'endpoint_url': 'http://my-url:my-port'
    },
}
file_system = s3fs.S3FileSystem(**storage_options)
file_system.open('s3://my-bucket/my_file.txt', 'r')

The output is:

FileNotFoundError: my-bucket/my_file.txt

And my HTTP traffic monitor shows the GET request for the list operation (note the list-type=2):

GET     /my-bucket?list-type=2&prefix=my_file.txt%2F&delimiter=%2F&max-keys=1&encoding-type=url

Anything else we need to know?:
The list operation seems to be requested when this line is executed.

Environment:

  • Python version: 3.6.9
  • Operating System: Ubuntu 18.04.3 LTS
  • Install method (conda, pip, source): pip install s3fs
  • Output of pip list:
Package           Version
----------------- -------
aiobotocore       1.1.2
aiohttp           3.6.2
aioitertools      0.7.0
async-timeout     3.0.1
attrs             20.2.0
botocore          1.17.44
chardet           3.0.4
docutils          0.15.2
fsspec            0.8.3
idna              2.10
idna-ssl          1.1.0
jmespath          0.10.0
multidict         4.7.6
pip               20.2.3
pkg-resources     0.0.0
python-dateutil   2.8.1
s3fs              0.5.1
setuptools        50.3.0
six               1.15.0
typing-extensions 3.7.4.3
urllib3           1.25.10
wheel             0.35.1
wrapt             1.12.1
yarl              1.6.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the referenced fsspec/spec.py lines 1212-1213 and reproduce the S3FileSystem.open example while monitoring requests. Trace why the missing-file path performs a list operation, then verify that it raises FileNotFoundError without issuing that request.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.