[Python] Python hangs when use pyarrow.fs.copy_files with "used_threads=True"
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
When try to copy a local path to s3 remote filesystem using `pyarrow.fs.copy_files` and using default parameter `use_threads=True`, the system hangs. If use "use_threads=False` the operation must complete ok (but more slow).
My code is:
```java
>>> import pyarrow as pa
>>> s3fs=pa.fs.S3FileSystem(endpoint_override="http://xxxxxx")
>>> pa.fs.copy_files("tests/data/payments", "bucket/payments", destination_filesystem=s3fs)
... (don't return)
```
If check remote s3, all files appear, but the function don't return
Platform: Windows
**Reporter**: [Alejandro Marco Ramos](https://issues.apache.org/jira/browse/ARROW-17064)
**Note**: *This issue was originally created as [ARROW-17064](https://issues.apache.org/jira/browse/ARROW-17064). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the pyarrow.fs.copy_files entry point and its use_threads handling for a local-to-S3FileSystem copy on Windows. Reproduce the hang with the supplied example, compare it with use_threads=False, and verify that the threaded operation returns after all files are copied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100