fsspec / fsspec/s3fs

improve error message of operations on a file if it is 'StorageClass': 'DEEP_ARCHIVE'

Open
#429 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

I was trying to get a file today and I realized I could not as it was 'StorageClass': 'DEEP_ARCHIVE'.

Doing
fs.get("s3://BUCKET/FILE", "FILE")
gave
PermissionError: The operation is not valid for the object's storage class
(full trackback below).

In doing fs.info("s3://BUCKET/FILE") I see 'StorageClass': 'DEEP_ARCHIVE'

It would be good if the error message added this as well e.g. "PermissionError: .... storage class is: {}".(fs.info("s3://BUCKET/FILE")['StorageClass'])

ClientError                               Traceback (most recent call last)
/opt/envs/shared/ds/lib/python3.8/site-packages/s3fs/core.py in _fetch_range(client, bucket, key, version_id, start, end, max_attempts, req_kw)
   1332         try:
-> 1333             resp = client.get_object(Bucket=bucket, Key=key,
   1334                                      Range='bytes=%i-%i' % (start, end - 1),

/opt/envs/shared/ds/lib/python3.8/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    315             # The "self" in this scope is referring to the BaseClient.
--> 316             return self._make_api_call(operation_name, kwargs)
    317 

/opt/envs/shared/ds/lib/python3.8/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    634             error_class = self.exceptions.from_code(error_code)
--> 635             raise error_class(parsed_response, operation_name)
    636         else:

ClientError: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class

During handling of the above exception, another exception occurred:

PermissionError                           Traceback (most recent call last)
<ipython-input-26-e6fe1581ad35> in <module>
----> 1 fs.get("s3://BUCKET/FILE", "FILE")

/opt/envs/shared/ds/lib/python3.8/site-packages/fsspec/spec.py in get(self, rpath, lpath, recursive, **kwargs)
    724         lpaths = other_paths(rpaths, lpath)
    725         for lpath, rpath in zip(lpaths, rpaths):
--> 726             self.get_file(rpath, lpath, **kwargs)
    727 
    728     def put_file(self, lpath, rpath, **kwargs):

/opt/envs/shared/ds/lib/python3.8/site-packages/fsspec/spec.py in get_file(self, rpath, lpath, **kwargs)
    704                     data = True
    705                     while data:
--> 706                         data = f1.read(self.blocksize)
    707                         f2.write(data)
    708 

/opt/envs/shared/ds/lib/python3.8/site-packages/fsspec/spec.py in read(self, length)
   1430             # don't even bother calling fetch
   1431             return b""
-> 1432         out = self.cache._fetch(self.loc, self.loc + length)
   1433         self.loc += len(out)
   1434         return out

/opt/envs/shared/ds/lib/python3.8/site-packages/fsspec/caching.py in _fetch(self, start, end)
    341         ):
    342             # First read, or extending both before and after
--> 343             self.cache = self.fetcher(start, bend)
    344             self.start = start
    345         elif start < self.start:

/opt/envs/shared/ds/lib/python3.8/site-packages/s3fs/core.py in _fetch_range(self, start, end)
   1198 
   1199     def _fetch_range(self, start, end):
-> 1200         return _fetch_range(self.fs.s3, self.bucket, self.key, self.version_id, start, end, req_kw=self.req_kw)
   1201 
   1202     def _upload_chunk(self, final=False):

/opt/envs/shared/ds/lib/python3.8/site-packages/s3fs/core.py in _fetch_range(client, bucket, key, version_id, start, end, max_attempts, req_kw)
   1350                                                               'InvalidRange']:
   1351                 return b''
-> 1352             raise translate_boto_error(e)
   1353         except Exception as e:
   1354             if 'time' in str(e).lower():  # Actual exception type changes often

PermissionError: The operation is not valid for the object's storage class

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in s3fs/core.py at _fetch_range and follow the InvalidObjectState path shown in the traceback, including translate_boto_error. Done means the resulting PermissionError identifies the object's storage class, with relevant behavior checked against the existing test coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.