drivendataorg / drivendataorg/cloudpathlib

Implement smart_open instead of .open() to allow efficient streaming (saving/loading) of large files to cloud bucket.

Open
#264 4 comments 0 reactions 0 assignees View on GitHub
caching cloudfile
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

The current implementation of the .open methods consists of a local cache which is then synchronized with the cloud.

This method can be replaced by [smart_open](https://github.com/RaRe-Technologies/smart_open), to allow for a more efficient mechanism.

One can take inspiration from aws' [S3PathLib](https://github.com/aws-samples/s3pathlib-project), (however, that library handles boto session in a way that is not thread-safe, which has made me switch to this library).

Currently, I subclassed S3Path and implemented the aforementioned S3Pathlib's implementation as follows:

```python
from cloudpathlib import S3Path as BaseS3Path
import smart_open

# replace the .open method of S3Path with smart_open
class S3Path(BaseS3Path):
def open(
self,
mode="r",
buffering=-1,
encoding=None,
errors=None,
newline=None,
closefd=True,
opener=None,
ignore_ext=False,
compression=None,
api_kwargs: dict = None, # type: ignore
):
"""
Open S3Path as a file-liked object.
:return: a file-like object.
See https://github.com/RaRe-Technologies/smart_open for more info.
"""

kwargs = dict(
uri=self.as_uri(),
mode=mode,
buffering=buffering,
encoding=encoding,
errors=errors,
newline=newline,
closefd=closefd,
opener=opener,
transport_params={"client": self.client}
)
return smart_open.open(**kwargs)

def read_text(
self,
encoding="utf-8",
errors=None,
) -> str:
with self.open(
mode="r",
encoding=encoding,
errors=errors,

) as f:
return f.read()

def read_bytes(self, ) -> bytes:
with self.open(mode="rb") as f:
return f.read()

def write_text(
self,
data: str,
encoding="utf-8",
errors=None,
newline=None,

):
with self.open(
mode="w",
encoding=encoding,
errors=errors,
newline=newline,
) as f:
f.write(data)

def write_bytes(self, data: bytes):
with self.open(mode="wb") as f:
f.write(data)

```

However, I could also open a pull request to merge this with the cloudpath definition, since smart_open is cloud-agnostic.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.