fsspec / fsspec/filesystem_spec

Can't access data from S3 Buckets

Open
#1,195 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

import fsspec
s3_fs = fsspec.filesystem("s3",  key="xxxxxx', secret="xxxxxxxxx",
                         client_kwargs={
                        "region_name": 'xxxxxxx',
                    })
s3_fs.ls('s3://')

It lists all the buckets but when i use

s3_fs.ls('s3://Bucket_name')
It returns empty

Same bucket can be accessed with boto3 and all the contents of the bucket but using fsspec returns empty. Please help me solve this issue

If I try to read the file using fsspec.open it gives bad request error because file was not found

I have also tried s3fs it has the same issue

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the shown Python reproduction using fsspec.filesystem("s3") and compare s3_fs.ls("s3://") with s3_fs.ls("s3://Bucket_name"). Check the same bucket through boto3, then verify that the fsspec listing exposes the expected contents and that fsspec.open can read a file without the reported bad request.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.