[Python][R] GcsFilesystem can appear to hang for non permanent errors
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
GcsFileSystem will attempt to retry if it gets a non-permanent error (such as couldn't connect to server). That's fine, except: (1) the sleep call used by the retry doesn't seem to check for interrupts and (2) the default retry timeout is 15 minutes!
The following snippets will hang for 15 minutes if you run them and wait about 5 seconds before trying to do a keyboard interrupt (CTRL+C):
```bash
Rscript -e 'library(arrow); fs <- GcsFileSystem$create(endpoint_override="localhost:1234", anonymous=TRUE); fs$CreateDir("x")'
python -c 'from pyarrow.fs import GcsFileSystem; fs = GcsFileSystem(endpoint_override="localhost:1234", anonymous=True); fs.create_dir("x")'
```
**Reporter**: [Will Jones](https://issues.apache.org/jira/browse/ARROW-17020) / @wjones127
#### Related issues:
- [[R][C++] Allow cancelling long-running commands](https://github.com/apache/arrow/issues/27688) (relates to)
**Note**: *This issue was originally created as [ARROW-17020](https://issues.apache.org/jira/browse/ARROW-17020). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the behavior with the Python and R commands in the issue, then trace GcsFileSystem's retry path and its sleep call. Done means a keyboard interrupt cancels the operation promptly instead of waiting for the 15-minute default retry timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python, r
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100