adobe / adobe/helix-mediahandler
Abstract storage layer to support alternative backends (e.g. Azure Blob Storage)
- Dominant language
- JavaScript
- Stars
- 4
- Forks
- 3
- Avg merge
- 7h 53m
- Merged PRs (30d)
- 6
Description
## Context
We currently support AWS S3 and Cloudflare R2 as storage backends. R2 support is possible today only because R2 is S3-API-compatible — the same `@aws-sdk/client-s3` client is pointed at a different endpoint. This issue analyzes what it would take to support a non-S3-compatible backend (e.g. Azure Blob Storage) and estimates the effort.
## Current state
- **Storage code is isolated to one file, but not behind an interface.** `@aws-sdk/client-s3` and `@aws-sdk/lib-storage` are used only in `src/MediaHandler.js` (925 lines) — no other file touches storage. However, S3 calls are inlined directly inside the same methods that do hashing, dimension detection, filtering, and URI generation. There is no `Storage`/`BlobStore` interface:
- `#fetchMetadata` (line ~290) — `HeadObjectCommand`, S3 only (not mirrored to R2).
- `putMetaData` (line ~510) — `CopyObjectCommand` with `MetadataDirective: 'REPLACE'` (a self-copy trick to update metadata without re-uploading the body), fanned out to S3 **and** R2 in parallel via `Promise.allSettled`.
- `#put` (line ~675) — builds an `@aws-sdk/lib-storage` `Upload` (automatic multipart) per backend, duplicating the body stream with `PassThrough` when it isn't a Buffer, run in parallel across S3 + R2.
- `#spool` (line ~807) — downloads source data, then calls `#put`.
- Constructor (lines ~129–159) — creates two `S3Client` instances: one for AWS, one for R2 pointed at `https://.r2.cloudflarestorage.com`.
- **Azure would not be a drop-in the way R2 was.** R2's "just point the same S3-compatible SDK at a different endpoint" trick does not extend to Azure Blob Storage, which speaks a different protocol entirely. It would require a different SDK/client (`@azure/storage-blob`, `BlockBlobClient`) with different multipart/chunking semantics, different metadata header conventions (`x-ms-meta-*`, lowercase keys, vs. S3's `x-amz-meta-*`), and a different error shape (Azure `RestError.statusCode` vs. the AWS SDK's `e.$metadata?.httpStatusCode`, used at lines ~302 and ~539).
- **The dual-write pattern is hardcoded to exactly two targets, not generic.** Both `putMetaData` and `#put` assume "S3 + R2" specifically, including position-based error attribution (`result[0].status === 'rejected' ? 'S3' : 'R2'`). Supporting a third/alternate backend means editing this logic in multiple places, not registering a new provider in a list.
- **The public contract leaks storage vocabulary.** `MediaHandlerOptions` (`src/MediaHandler.d.ts`) exposes `awsRegion`, `awsAccessKeyId`, `awsSecretAccessKey`, `r2AccountId`, `r2AccessKeyId`, `r2SecretAccessKey`, `disableR2` directly to consumers, and `MediaResource.storageUri` is literally formatted as `s3://bucket/key` (line ~495). Staying backward compatible while adding a different backend means either keeping AWS/R2-flavored field names (misleading once Azure is involved) or making a breaking change to the options shape / bumping a major version.
- **Tests are wire-protocol-coupled, not interface-coupled.** `test/mediahandler.test.js` (~1800 lines) mocks S3 with `nock` at the raw HTTP level: literal S3 multipart XML bodies (`InitiateMultipartUploadResult`, `CompleteMultipartUploadResult`), S3 query-string command markers (`?uploads=&x-id=CreateMultipartUpload`), and `x-amz-*` request/response headers. None of this is reusable for a different backend — a parallel mock suite would be needed for Azure's REST protocol.
## Difficulty assessment: Medium
Not architecturally hard — there's only one file, and extracting a `StorageProvider`-style interface (`head(key)`, `put(key, body, meta)`, `updateMetadata(key, meta)`) so `MediaHandler` depends on an abstraction instead of `S3Client` directly is a mostly mechanical refactor. The real cost centers are:
1. Generalizing dual-write from a hardcoded S3+R2 pair to an N-provider list (or deciding Azure replaces one of them rather than joining them).
2. Redesigning the public options/typing contract (`MediaHandlerOptions`, `MediaResource.storageUri`) without breaking existing consumers, or accepting a major version bump.
3. Rewriting the test suite — the largest cost, since current tests assert on S3 wire format rather than on interface behavior.
4. Re-validating AWS-SDK-specific behaviors that don't map 1:1: multipart upload threshold semantics (the code comment at line ~836 notes "the s3 multipart uploader has a default min size of 5mb," which drove the `uploadBufferSize` design — Azure's `BlockBlobClient` chunking behaves differently).
One genuine simplification if Azure were adopted: `putMetaData`'s S3 self-copy metadata trick becomes unnecessary, since Azure Blob has a direct `setMetadata()` call.
## Suggested approach (not scoped/estimated here, for discussion)
Introduce a small storage interface implemented by an `S3StorageProvider` (wrapping today's logic, generalized to a list of mirrored targets instead of a hardcoded S3+R2 pair) and, if needed, an `AzureBlobStorageProvider`. `MediaHandler` would hold a list of providers and stop importing `@aws-sdk/*` directly. This isolates the mechanical refactor from the two real decisions the team needs to make first: whether the public `MediaHandlerOptions`/`MediaResource` contract can change, and whether Azure replaces the R2 mirror or becomes a third target.
Contributor guide
Research direction
Start with src/MediaHandler.js and src/MediaHandler.d.ts, then inspect test/mediahandler.test.js to understand the current S3-specific behavior and public contract. Resolve whether the abstraction supports mirrored providers and whether Azure replaces or joins them; done means the provider design, compatibility approach, and wire-protocol test strategy are agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, azure, javascript
- Domain
- backend, cloud
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100