Using s3fs with no 'LastModified' key info

Open
#162 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
cloud

Research direction

Start with s3fs/core.py at the info() code reported at line 447, then follow how open() obtains object metadata. Reproduce the Ceph response without LastModified and compare it with the metadata expected by S3FileSystem. Done means the reported open() operation handles this response without the KeyError while preserving existing ls(), mkdir(), isfile(), and isdir() behavior.

Written by the indexing model from the issue text.

Description

I have a Ceph based Object store with S3, and I am trying to use s3fs.

The metadata of the objects do not seem to have 'LastModified' entry, which was identified using Boto3's s3.Client.head_object().
Here's a sample metadata response of one of the objects

{'ResponseMetadata': {'RequestId': 'tx0000000000000067d7a8a-005c3dce53-489594c6-in-chennai', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'openresty/1.13.6.2', 'date': 'Tue, 15 Jan 2019 12:13:07 GMT', 'content-type': 'application/octet-stream', 'accept-ranges': 'bytes', 'etag': '"ed076287532e86365e841e92bfc50d8c"', 'content-length': '12', 'x-amz-request-id': 'tx0000000000000067d7a8a-005c3dce53-489594c6-in-chennai', 'expires': 'Thu, 31 Dec 2037 23:55:55 GMT', 'cache-control': 'max-age=315360000', 'x-e-id': '10.34.34.118', 'x-elb-id': '10.34.34.118', 'x-elb-app-server': '10.34.105.188'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 'ContentLength': 12, 'ETag': '"ed076287532e86365e841e92bfc50d8c"', 'CacheControl': 'max-age=315360000', 'ContentType': 'application/octet-stream', 'Expires': datetime.datetime(2037, 12, 31, 23, 55, 55, tzinfo=tzutc()), 'Metadata': {}}

When I try to use s3fs operations like open() [To open a file for reading and writing], it gives the following error

Traceback (most recent call last):
  File "/Users/aman.garg/Downloads/test/s3fs-touch.py", line 11, in <module>
    with fs.open('amantestnotebook/notebook/write_file.txt', 'rb') as f:
  File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 315, in open
    s3_additional_kwargs=kw)
  File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 1102, in __init__
    info = self.info()
  File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 1120, in info
    refresh=refresh, **kwargs)
  File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 447, in info
    'LastModified': out['LastModified'],
KeyError: 'LastModified'

Operations like ls(), mkdir(),isfile(), isdir() are working as expected.

Is there a way to use S3FileSystem without the 'LastModified' metadata? What exactly is this LastModified entry used for?

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fsspec/s3fs

All issues in fsspec/s3fs

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.