apache / apache/arrow

[Python] Retry_strategy is not included in S3FileSystem.__reduce__ method

Open
#47,348 0 comments 0 reactions 1 assignee Claimed by @wingkitlee0 View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

In some distributed processing libraries like Ray, a `pyarrow.fs.S3FileSystem` is serialized and sent to different workers.

https://github.com/ray-project/ray/blob/a1b9dc627c7ab0b31ae5df03892f4d49192463e4/python/ray/data/datasource/file_based_datasource.py#L485

As the retry_strategy is not included in `__reduce__`, the copies do not retry s3 calls:

https://github.com/apache/arrow/blob/4636e8a01d41e88b4cf3a44e15c9874211cecadb/python/pyarrow/_s3fs.pyx#L452-L484

Currently it's not easy to reproduce this in a standalone script as retry_strategy and self.s3fs is not accessible outside cython. Maybe someone can comment/verify if this makes sense and I can try to debug further.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.