[Python] Retry_strategy is not included in S3FileSystem.__reduce__ method
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
In some distributed processing libraries like Ray, a `pyarrow.fs.S3FileSystem` is serialized and sent to different workers.
https://github.com/ray-project/ray/blob/a1b9dc627c7ab0b31ae5df03892f4d49192463e4/python/ray/data/datasource/file_based_datasource.py#L485
As the retry_strategy is not included in `__reduce__`, the copies do not retry s3 calls:
https://github.com/apache/arrow/blob/4636e8a01d41e88b4cf3a44e15c9874211cecadb/python/pyarrow/_s3fs.pyx#L452-L484
Currently it's not easy to reproduce this in a standalone script as retry_strategy and self.s3fs is not accessible outside cython. Maybe someone can comment/verify if this makes sense and I can try to debug further.
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.