[C++] S3 FileSystem doesn't seem to handle redirects
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
In some conditions AWS S3 seems to respond with a redirect, but Arrow seems to consider it an error instead of following the redirect.
For example see
```Java
s3, bucket = fs.FileSystem.from_uri("s3://ursa-labs-taxi-data/?region=us-east-1")
print(s3.get_file_info(fs.FileSelector(bucket+"/2011", recursive=True)))
```
The error that you get is
```Java
OSError: When listing objects under key '2011' in bucket 'ursa-labs-taxi-data': AWS Error [code 100]: Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
```
It should probably follow the `PermanentRedirect` instead of choking over it
IT is also possible to reproduce it using
```Java
from pyarrow import fs
s3 = fs.SubTreeFileSystem("ursa-labs-taxi-data", fs.S3FileSystem())
print(s3.get_file_info(fs.FileSelector("2011", recursive=True)))
```
**Reporter**: [Alessandro Molina](https://issues.apache.org/jira/browse/ARROW-13237) / @amol-
#### Related issues:
- [[Python] Expose S3 region resolver](https://github.com/apache/arrow/issues/30669) (relates to)
**Note**: *This issue was originally created as [ARROW-13237](https://issues.apache.org/jira/browse/ARROW-13237). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the `PermanentRedirect` with either S3 example in the issue: the `FileSystem.from_uri` listing or the `SubTreeFileSystem` listing. Read how the C++ S3 filesystem handles listing errors and redirects. Done means the redirect is followed and the listing succeeds rather than returning the shown error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100