Using s3fs with no 'LastModified' key info
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
Research direction
Start with s3fs/core.py at the info() code reported at line 447, then follow how open() obtains object metadata. Reproduce the Ceph response without LastModified and compare it with the metadata expected by S3FileSystem. Done means the reported open() operation handles this response without the KeyError while preserving existing ls(), mkdir(), isfile(), and isdir() behavior.
Written by the indexing model from the issue text.
Description
I have a Ceph based Object store with S3, and I am trying to use s3fs.
The metadata of the objects do not seem to have 'LastModified' entry, which was identified using Boto3's s3.Client.head_object().
Here's a sample metadata response of one of the objects
{'ResponseMetadata': {'RequestId': 'tx0000000000000067d7a8a-005c3dce53-489594c6-in-chennai', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'openresty/1.13.6.2', 'date': 'Tue, 15 Jan 2019 12:13:07 GMT', 'content-type': 'application/octet-stream', 'accept-ranges': 'bytes', 'etag': '"ed076287532e86365e841e92bfc50d8c"', 'content-length': '12', 'x-amz-request-id': 'tx0000000000000067d7a8a-005c3dce53-489594c6-in-chennai', 'expires': 'Thu, 31 Dec 2037 23:55:55 GMT', 'cache-control': 'max-age=315360000', 'x-e-id': '10.34.34.118', 'x-elb-id': '10.34.34.118', 'x-elb-app-server': '10.34.105.188'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 'ContentLength': 12, 'ETag': '"ed076287532e86365e841e92bfc50d8c"', 'CacheControl': 'max-age=315360000', 'ContentType': 'application/octet-stream', 'Expires': datetime.datetime(2037, 12, 31, 23, 55, 55, tzinfo=tzutc()), 'Metadata': {}}
When I try to use s3fs operations like open() [To open a file for reading and writing], it gives the following error
Traceback (most recent call last):
File "/Users/aman.garg/Downloads/test/s3fs-touch.py", line 11, in <module>
with fs.open('amantestnotebook/notebook/write_file.txt', 'rb') as f:
File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 315, in open
s3_additional_kwargs=kw)
File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 1102, in __init__
info = self.info()
File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 1120, in info
refresh=refresh, **kwargs)
File "/Users/aman.garg/anaconda2/envs/s3contents/lib/python3.7/site-packages/s3fs/core.py", line 447, in info
'LastModified': out['LastModified'],
KeyError: 'LastModified'
Operations like ls(), mkdir(),isfile(), isdir() are working as expected.
Is there a way to use S3FileSystem without the 'LastModified' metadata? What exactly is this LastModified entry used for?
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from fsspec/s3fs
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 50/100
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100