drivendataorg / drivendataorg/cloudpathlib
Implement smart_open instead of .open() to allow efficient streaming (saving/loading) of large files to cloud bucket.
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
The current implementation of the .open methods consists of a local cache which is then synchronized with the cloud.
This method can be replaced by [smart_open](https://github.com/RaRe-Technologies/smart_open), to allow for a more efficient mechanism.
One can take inspiration from aws' [S3PathLib](https://github.com/aws-samples/s3pathlib-project), (however, that library handles boto session in a way that is not thread-safe, which has made me switch to this library).
Currently, I subclassed S3Path and implemented the aforementioned S3Pathlib's implementation as follows:
```python
from cloudpathlib import S3Path as BaseS3Path
import smart_open
# replace the .open method of S3Path with smart_open
class S3Path(BaseS3Path):
def open(
self,
mode="r",
buffering=-1,
encoding=None,
errors=None,
newline=None,
closefd=True,
opener=None,
ignore_ext=False,
compression=None,
api_kwargs: dict = None, # type: ignore
):
"""
Open S3Path as a file-liked object.
:return: a file-like object.
See https://github.com/RaRe-Technologies/smart_open for more info.
"""
kwargs = dict(
uri=self.as_uri(),
mode=mode,
buffering=buffering,
encoding=encoding,
errors=errors,
newline=newline,
closefd=closefd,
opener=opener,
transport_params={"client": self.client}
)
return smart_open.open(**kwargs)
def read_text(
self,
encoding="utf-8",
errors=None,
) -> str:
with self.open(
mode="r",
encoding=encoding,
errors=errors,
) as f:
return f.read()
def read_bytes(self, ) -> bytes:
with self.open(mode="rb") as f:
return f.read()
def write_text(
self,
data: str,
encoding="utf-8",
errors=None,
newline=None,
):
with self.open(
mode="w",
encoding=encoding,
errors=errors,
newline=newline,
) as f:
f.write(data)
def write_bytes(self, data: bytes):
with self.open(mode="wb") as f:
f.write(data)
```
However, I could also open a pull request to merge this with the cloudpath definition, since smart_open is cloud-agnostic.
Contributor guide
Assessment
This issue has not been assessed yet.