[Python] `S3FileSystem` does not appear to respect `HTTP_PROXY` or `HTTPS_PROXY` environment variables
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
It would appear that the `S3FileSystem` does not respect the proxy variables, and requires usage of the `proxy_options` argument. If this is expected behavior, I would argue that it should default to the environment variables if present.
```python
>>> import pyarrow
>>> pyarrow.__version__
'21.0.0'
>>> from pyarrow import fs
>>> system = fs.S3FileSystem()
>>> system.get_file_info(file)
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/_fs.pyx", line 615, in pyarrow._fs.FileSystem.get_file_info
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: When getting information for key *** in bucket ***: AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 35, SSL connect error
>>> import os
>>> proxy = os.environ['HTTP_PROXY']
>>> system = fs.S3FileSystem(proxy_options=proxy)
>>> system.get_file_info(file)
```
My work restricts access to external resources, and we leverage the usage of `HTTP_PROXY` and `HTTPS_PROXY` to work around this restriction when necessary. PyArrow is the first package I have come across that does not respect these proxy variables by default, so I assumed it was convention that they would be used if present.
Note: aws credentials, profile, region, etc. are set elsewhere, as expected, within`~/.aws/config` and `~/.aws/credentials`
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.