apache / apache/arrow

[Python] Python hangs when use pyarrow.fs.copy_files with "used_threads=True"

Open
#32,372 5 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

When try to copy a local path to s3 remote filesystem using `pyarrow.fs.copy_files` and using default parameter `use_threads=True`, the system hangs. If use "use_threads=False` the operation must complete ok (but more slow).

 

My code is:
```java

>>> import pyarrow as pa
>>> s3fs=pa.fs.S3FileSystem(endpoint_override="http://xxxxxx")
>>> pa.fs.copy_files("tests/data/payments", "bucket/payments", destination_filesystem=s3fs)
... (don't return)
```
If check remote s3, all files appear, but the function don't return

 

Platform: Windows

**Reporter**: [Alejandro Marco Ramos](https://issues.apache.org/jira/browse/ARROW-17064)

**Note**: *This issue was originally created as [ARROW-17064](https://issues.apache.org/jira/browse/ARROW-17064). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the pyarrow.fs.copy_files entry point and its use_threads handling for a local-to-S3FileSystem copy on Windows. Reproduce the hang with the supplied example, compare it with use_threads=False, and verify that the threaded operation returns after all files are copied.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.