apache / apache/opendal

bug: python - S3 operator.open (opendal.File) hangs indefinitely while trying to write pyarrow table

Open
#6,373 2 comments 1 reaction 0 assignees View on GitHub
bindings/python bug services/s3
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Describe the bug

Hello!

I'm having some trouble using the operator (Python bindings) to write a pyarrow table to S3. I'm using `operator.open` (so an `opendal.File`) as a context manager and passing this object to `pyarrow.parquet.ParquetWriter` along with my table to be persisted to S3. When calling `write_table` on the writer object in a unit test, the operation hangs indefinitely (no failure).

Reading a table from S3 works as expected, it's writing that's causing us trouble.

### Steps to Reproduce

```python
from pathlib import Path

import boto3
import pyarrow as pa
from moto import mock_aws
from moto.moto_server.threaded_moto_server import ThreadedMotoServer
from opendal import Operator

def test_s3_operator_write() -> None:
with mock_aws():
server = ThreadedMotoServer(port=0)
server.start()
_, port = server.get_host_and_port()
endpoint_url = f"http://localhost:{port}"

conn = boto3.resource("s3", endpoint_url=endpoint_url)
conn.create_bucket(Bucket="test-bucket")

handler = Operator(
"s3",
bucket="test-bucket",
region="auto",
endpoint=endpoint_url # Add the mock endpoint URL
)

table = pa.Table.from_pydict({"A": [2]})
dest = Path("prefix/table.parquet")
with handler.open(dest, mode="wb") as f:
with pa.parquet.ParquetWriter(f) as writer:
writer.write(table)

server.stop()
```

### Expected Behavior

The test would actually execute the write to S3, and then I would verify this by reading using the operator.

### Additional Context

I tried wrapping the operator in `pyarrow.PythonFile` and passing it to `ParquetWriter`, but to no avail.

I wonder if I'm doing something incorrectly, or if this is even expected behaviour given [this comment](https://github.com/apache/opendal/issues/4363) about file-like operations being supported only for `fs` mode?

Would appreciate your help, thanks!

### Are you willing to submit a PR to fix this bug?

- [ ] Yes, I would like to submit a PR.

Contributor guide

Open the contributing guide

Research direction

Start with the provided reproduction using Operator.open, opendal.File, and pyarrow.parquet.ParquetWriter, then inspect the Python binding's S3 write path and the file-like-operation limitation referenced in issue #4363. Done means the write completes for the shown S3 setup and the resulting table can be read back with the operator.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.