invoke-ai / invoke-ai/InvokeAI
[enhancement]: Add S3-compatible ImageFileStorage (first cloud impl of ImageFileStorageBase)
- Dominant language
- Python
- Stars
- 28.2k
- Forks
- 3k
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 19
Description
Concrete cloud implementation of `ImageFileStorageBase`. The ABC was introduced in #1650 (2023) where @Kyle0654 explicitly named cloud storage as the design rationale: *"if someone wants to use cloud storage for their images, they should be able to replace the image storage service easily."* @psychedelicious echoed this on #1425 in 2022.
**Proposal**
`S3CompatibleImageFileStorage(ImageFileStorageBase)` at `invokeai/app/services/image_files/image_files_s3.py`, parallel to `image_files_disk.py`. boto3 only — no new runtime deps. Selected via `InvokeAIAppConfig.storage_backend: Literal["disk", "s3"]` (default `disk`) with `s3_bucket` and `s3_endpoint_url` companions. Works against AWS S3 by default; `s3_endpoint_url` makes the same backend work against any S3-compatible store. For Backblaze B2, `B2_APPLICATION_KEY_ID` / `B2_APPLICATION_KEY` are aliased onto AWS-named credentials at client-construction time.
**Open questions** (would like sign-off before final push):
1. **LRU cache** — disk has one (`__cache`); S3 impl is stateless. Read-through cache now, or v1 stateless + cache as a follow-up?
2. **`pil_compress_level`** — disk reads it from config at save time; S3 currently uses PIL's default. Thread it through?
3. **`get_path` contract** — the ABC types it as `Path` for a `FileResponse` flow that doesn't apply to S3. OK to return a synthetic `s3://bucket/key` Path until a presigned-URL service follow-up?
4. **moto** — not in dev deps; tests use a hand-rolled fake S3 client. Add moto for closer-to-real round-trips, or keep deps minimal?
5. **Follow-up scope** — `S3CompatibleObjectSerializer` (latents) and `S3PresignedUrlService` (frontend direct-fetch) as separate PRs, or folded into this one?
**Draft branch:** [goanpeca:enh/s3-image-file-storage](https://github.com/goanpeca/InvokeAI/tree/enh/s3-image-file-storage) — IO class + 10 unit tests + 3 config tests + dispatch wire-up + docs page. Lint clean.
### Alternatives
- **FUSE mount (rclone, goofys, s3fs-fuse)** — breaks the abstraction, worse latency for small reads, no presigned-URL path.
- **Out-of-tree plugin** — viable, but the disk impl is in tree; in-tree S3 keeps parity, discoverability, and a single test surface.
- **Wait for a community contribution** — #1425 was filed in 2022; three years and counting.
### Additional Content
Precedent in tree:
- `invokeai/app/services/image_files/image_files_base.py` — ABC
- `invokeai/app/services/image_files/image_files_disk.py` — only prior impl; new file mirrors its shape
- `invokeai/app/services/object_serializer/object_serializer_disk.py` — pattern for `S3CompatibleObjectSerializer` follow-up
- `invokeai/app/services/urls/urls_default.py` — pattern for `S3PresignedUrlService` follow-up
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.