apache / apache/arrow

[Python] `S3FileSystem` does not appear to respect `HTTP_PROXY` or `HTTPS_PROXY` environment variables

Open
#47,805 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

It would appear that the `S3FileSystem` does not respect the proxy variables, and requires usage of the `proxy_options` argument. If this is expected behavior, I would argue that it should default to the environment variables if present.

```python
>>> import pyarrow
>>> pyarrow.__version__
'21.0.0'

>>> from pyarrow import fs
>>> system = fs.S3FileSystem()
>>> system.get_file_info(file)
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/_fs.pyx", line 615, in pyarrow._fs.FileSystem.get_file_info
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: When getting information for key *** in bucket ***: AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 35, SSL connect error

>>> import os
>>> proxy = os.environ['HTTP_PROXY']
>>> system = fs.S3FileSystem(proxy_options=proxy)
>>> system.get_file_info(file)

```

My work restricts access to external resources, and we leverage the usage of `HTTP_PROXY` and `HTTPS_PROXY` to work around this restriction when necessary. PyArrow is the first package I have come across that does not respect these proxy variables by default, so I assumed it was convention that they would be used if present.

Note: aws credentials, profile, region, etc. are set elsewhere, as expected, within`~/.aws/config` and `~/.aws/credentials`
### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.