bug: python - S3 operator.open (opendal.File) hangs indefinitely while trying to write pyarrow table
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Describe the bug
Hello!
I'm having some trouble using the operator (Python bindings) to write a pyarrow table to S3. I'm using `operator.open` (so an `opendal.File`) as a context manager and passing this object to `pyarrow.parquet.ParquetWriter` along with my table to be persisted to S3. When calling `write_table` on the writer object in a unit test, the operation hangs indefinitely (no failure).
Reading a table from S3 works as expected, it's writing that's causing us trouble.
### Steps to Reproduce
```python
from pathlib import Path
import boto3
import pyarrow as pa
from moto import mock_aws
from moto.moto_server.threaded_moto_server import ThreadedMotoServer
from opendal import Operator
def test_s3_operator_write() -> None:
with mock_aws():
server = ThreadedMotoServer(port=0)
server.start()
_, port = server.get_host_and_port()
endpoint_url = f"http://localhost:{port}"
conn = boto3.resource("s3", endpoint_url=endpoint_url)
conn.create_bucket(Bucket="test-bucket")
handler = Operator(
"s3",
bucket="test-bucket",
region="auto",
endpoint=endpoint_url # Add the mock endpoint URL
)
table = pa.Table.from_pydict({"A": [2]})
dest = Path("prefix/table.parquet")
with handler.open(dest, mode="wb") as f:
with pa.parquet.ParquetWriter(f) as writer:
writer.write(table)
server.stop()
```
### Expected Behavior
The test would actually execute the write to S3, and then I would verify this by reading using the operator.
### Additional Context
I tried wrapping the operator in `pyarrow.PythonFile` and passing it to `ParquetWriter`, but to no avail.
I wonder if I'm doing something incorrectly, or if this is even expected behaviour given [this comment](https://github.com/apache/opendal/issues/4363) about file-like operations being supported only for `fs` mode?
Would appreciate your help, thanks!
### Are you willing to submit a PR to fix this bug?
- [ ] Yes, I would like to submit a PR.
Contributor guide
Research direction
Start with the provided reproduction using Operator.open, opendal.File, and pyarrow.parquet.ParquetWriter, then inspect the Python binding's S3 write path and the file-like-operation limitation referenced in issue #4363. Done means the write completes for the shown S3 setup and the resulting table can be read back with the operator.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100