drivendataorg / drivendataorg/cloudpathlib
How to use cloudpathlib without any cache or disk I/O?
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
I'm using cloudpathlib for file write, read, delete as part of a web app.
Running it on AWS AppRunner and S3 we are seeing some issues that sometimes delete doesn't work. cloudpathlib claims to have deleted the file but then the next GET request shows it's still there.
Could this be a cloudpathlib caching issue?
I would like to turn caching off completely and also not use a local disk or tempfiles. It should be just a stateless in-memory HTTP application server.
Is this possible? How do I configure cloudpathlib to NOT have caching and NOT have any disk I/O?
This is the utils `s3.py` module I'm using at the moment:
```python
"""File storage utilities for handling both S3 and local filesystem operations."""
import base64
import logging
from cloudpathlib import S3Client, S3Path
from fastapi import UploadFile
from stocadro.config import settings
from stocadro.core.schemas import MemFile
logger = logging.getLogger(__name__)
# S3 client configuration
s3_client = S3Client(
aws_access_key_id=settings.S3_ACCESS_KEY_ID,
aws_secret_access_key=settings.S3_ACCESS_KEY,
)
s3_client.set_as_default_client()
s3_image_path = S3Path(settings.s3_image_path)
class S3:
"""File storage on S3 or local filesystem with uniform interface."""
@staticmethod
def write(input_file: UploadFile, filename: str) -> None:
path = s3_image_path / filename
logger.info(f"S3 write {path=}")
path.parent.mkdir(exist_ok=True, parents=True)
path.write_bytes(input_file.file.read())
@staticmethod
def read(filename: str) -> MemFile:
path = s3_image_path / filename
logger.info(f"S3 read {path=}")
return MemFile(content=path.read_bytes(), name=filename)
@staticmethod
def delete(filename: str) -> None:
path = s3_image_path / filename
logger.info(f"S3 delete: {path=}")
path.unlink()
```
Contributor guide
Assessment
This issue has not been assessed yet.