[Python] Cannot call pyarrow.fs.initialize_s3
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Steps to reproduce:
```
>>> import pyarrow.fs
>>> pyarrow.fs.initialize_s3(pyarrow.fs.S3LogLevel.Debug)
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/_s3fs.pyx", line 57, in pyarrow._s3fs.initialize_s3
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: S3 was already initialized. It is safe to use but the options passed in this call have been ignored.
```
Cause:
ensure_s3_initialized() is called when pyarrow.fs is imported. This call itself is somewhat slow, so users may want to defer calling it until they actually initialize an S3FileSystem class.
https://github.com/apache/arrow/blob/cd6e2a4d2b9373b942da18b4cc82cb41431764d9/python/pyarrow/fs.py#L61
Workaround:
Import the '_s3fs' module first:
```
>>> import pyarrow._s3fs
>>> pyarrow._s3fs.initialize_s3(pyarrow._s3fs.S3LogLevel.Debug)
```
Version:
```
pip list | grep pyarrow
pyarrow 12.0.0
```
### Component(s)
Python
Contributor guide
Research direction
Start in python/pyarrow/fs.py at the ensure_s3_initialized() call around line 61, then trace the initialize_s3 and S3FileSystem entry points. Confirm the import-time initialization path and the explicit initialization behavior; done means importing pyarrow.fs does not prevent initialize_s3 from applying its options.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- api, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100