fsspec / fsspec/s3fs

Does aioboto3 Support Authentication with EC2 IAM Roles?

Open
#874 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

Describe the bug
While using the boto3 library, I can authenticate and interact with AWS services using the IAM role assigned to my EC2 instance. However, when attempting to perform similar operations with s3fs, I encounter errors. Below is the code snippet demonstrating the behavior with boto3 and the corresponding error with s3fs.

Environment:

  • Python Version: 3.10
  • OS name and version: ubuntu
  • s3fs-2024.3.1

Sample code

import boto3

# Create boto3 client
s3 = boto3.client('s3')

# List contents of a bucket
response = s3.list_objects_v2(Bucket='your-bucket-name')
print(response)

import s3fs
fs = s3fs.S3FileSystem(anon=False)
# Attempt to list the same bucket with s3fs, expecting to use the same IAM role
print(fs.ls('your-bucket-name'))

Error log

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 720, in _lsdir
    async for c in self._iterdir(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 770, in _iterdir
    async for i in it:
  File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/paginate.py", line 30, in __anext__
    response = await self._make_request(current_kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/client.py", line 408, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: No AWSAccessKey was presented.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 1005, in _ls
    files = await self._lsdir(path, refresh, versions=versions)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 733, in _lsdir
    raise translate_boto_error(e)
PermissionError: No AWSAccessKey was presented.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported calls in s3fs/core.py, especially _lsdir, _iterdir, and _ls, and compare S3FileSystem(anon=False) with the working boto3 client. Reproduce the EC2 IAM-role case using the provided sample and verify whether fs.ls can list the bucket without an access key; document or fix the behavior based on that result.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.