apache / apache/arrow

[Python] Cannot call pyarrow.fs.initialize_s3

Open
#35,575 8 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Steps to reproduce:
```
>>> import pyarrow.fs
>>> pyarrow.fs.initialize_s3(pyarrow.fs.S3LogLevel.Debug)
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/_s3fs.pyx", line 57, in pyarrow._s3fs.initialize_s3
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: S3 was already initialized. It is safe to use but the options passed in this call have been ignored.
```

Cause:
ensure_s3_initialized() is called when pyarrow.fs is imported. This call itself is somewhat slow, so users may want to defer calling it until they actually initialize an S3FileSystem class.

https://github.com/apache/arrow/blob/cd6e2a4d2b9373b942da18b4cc82cb41431764d9/python/pyarrow/fs.py#L61

Workaround:
Import the '_s3fs' module first:
```
>>> import pyarrow._s3fs
>>> pyarrow._s3fs.initialize_s3(pyarrow._s3fs.S3LogLevel.Debug)
```

Version:
```
pip list | grep pyarrow
pyarrow 12.0.0
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start in python/pyarrow/fs.py at the ensure_s3_initialized() call around line 61, then trace the initialize_s3 and S3FileSystem entry points. Confirm the import-time initialization path and the explicit initialization behavior; done means importing pyarrow.fs does not prevent initialize_s3 from applying its options.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
api, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.