drivendataorg / drivendataorg/cloudpathlib

Compatibility with `aiofiles`

Open
#376 1 comment 2 reactions 0 assignees View on GitHub
caching cloudfile
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

`aiofiles` is a Python project to perform IO asynchronously. I am able to asynchronously read files from s3, but I cannot write files to s3. The following code example fails:

```
import asyncio

import aiofiles
import aiohttp
from cloudpathlib import S3Path

async def main():
chunk_size = 1000

path = S3Path("s3://my-bucket/image.png")

async with aiohttp.ClientSession() as session:
async with session.get("http://httpbin.org/image/png") as resp:
async with aiofiles.open(path, "wb") as f:
async for chunk in resp.content.iter_chunked(chunk_size):
await f.write(chunk)

asyncio.run(main())
```
The error is:
```
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpq4ls9jhw/....../image.png'
```
I am using `cloudpathlib==0.16.0` and `aiofiles==23.2.1`.

If I add a `path.touch()` before initializing the `ClientSession`, the code runs without error, but the file in s3 is zero bytes.

Thank you for your work on such a clean and easy to use library! It truly is a joy!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.