apache / apache/arrow

[C++] S3 FileSystem doesn't seem to handle redirects

Open
#28,922 6 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

In some conditions AWS S3 seems to respond with a redirect, but Arrow seems to consider it an error instead of following the redirect.

For example see

```Java

s3, bucket = fs.FileSystem.from_uri("s3://ursa-labs-taxi-data/?region=us-east-1")
print(s3.get_file_info(fs.FileSelector(bucket+"/2011", recursive=True)))
```

The error that you get is

```Java

OSError: When listing objects under key '2011' in bucket 'ursa-labs-taxi-data': AWS Error [code 100]: Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
```

It should probably follow the `PermanentRedirect` instead of choking over it

IT is also possible to reproduce it using
```Java

from pyarrow import fs

s3 = fs.SubTreeFileSystem("ursa-labs-taxi-data", fs.S3FileSystem())
print(s3.get_file_info(fs.FileSelector("2011", recursive=True)))
```

**Reporter**: [Alessandro Molina](https://issues.apache.org/jira/browse/ARROW-13237) / @amol-
#### Related issues:
- [[Python] Expose S3 region resolver](https://github.com/apache/arrow/issues/30669) (relates to)

**Note**: *This issue was originally created as [ARROW-13237](https://issues.apache.org/jira/browse/ARROW-13237). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the `PermanentRedirect` with either S3 example in the issue: the `FileSystem.from_uri` listing or the `SubTreeFileSystem` listing. Read how the C++ S3 filesystem handles listing errors and redirects. Done means the redirect is followed and the listing succeeds rather than returning the shown error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.