adobe / adobe/helix-shared

Add helix-shared-queue base package (pluggable queue abstraction)

Open
#1,269 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
15
Forks
16
Avg merge
23h 18m
Merged PRs (30d)
14

Description

## Why

helix-api-service (and likely other Helix services) send messages to AWS SQS via `BatchedQueueClient` (`@adobe/helix-admin-support`), which hardcodes `@aws-sdk/client-sqs` + `@aws-sdk/client-s3` internally. There's no way to back it with a different queue provider (e.g. Azure Service Bus) without forking that class.

`@adobe/helix-shared-storage` already solves this exact problem for blob storage: a thin base package with a pluggable `backendFactory`, consumed today via `@adobe/helix-shared-storage-s3`. This issue proposes the same pattern for queues.

## Proposed interface

Mirrors `Storage`/`Bucket`:

- `Queue.fromContext(context, opts)` — caches on `context.attributes.queue`, calls `new this(...)` so subclasses compose correctly (see `Storage.fromContext`).
- `Queue#queue(queueName, opts)` -> `QueueRef`, constructed via an injected `backendFactory` (no cloud SDK in this package at all).
- `QueueRef#send(messages)` — `messages: [{ body, groupId?, dedupId? }]` -> `{ messageIds }`. `groupId`/`dedupId` are named generically (not `MessageGroupId`/`MessageDeduplicationId`) so they can also map onto Azure Service Bus `sessionId` + native dedup later.
- `QueueRef#receive(opts)` / `QueueRef#delete(messages)` — generalizes `BatchedQueueClient.receive()`/`delete()` (long-poll semantics, batch ack), for consumers of these queues outside helix-api-service.
- `AbstractQueueBackend` — the contract concrete packages implement (`sendBatch`/`receiveBatch`/`deleteBatch`). Batching/chunking limits are provider-specific (SQS's ≤10 entries/≤256KB vs. Azure Service Bus's dynamic batch sizing), so each backend owns its own batching loop; `QueueRef` is a thin pass-through, not a shared generic chunker.

## Notable design choice: reuse `@adobe/helix-shared-storage` for oversized-message spillover

`BatchedQueueClient` hardcodes S3 (`PutObjectCommand`) for its "message too big for the queue, swap to blob storage" fallback. Backends built on this package should instead accept an injected `Storage`/`Bucket` (from `@adobe/helix-shared-storage`) for that fallback, so the spill-storage choice tracks whatever `Storage` backend the host service already configured, rather than each queue backend hardcoding its own cloud SDK for a secondary concern.

## Out of scope here

- No concrete backend (see companion issue for `helix-shared-queue-sqs`).
- Consumer-side "trigger a function from a queue" wiring (e.g. Lambda SQS event sources) is a runtime/deployment concern, not part of this package.

Contributor guide

Open the contributing guide

Research direction

Start by studying the existing Storage.fromContext, Storage, and Bucket patterns in @adobe/helix-shared-storage, then compare the queue behavior described for BatchedQueueClient. Define the base Queue, QueueRef, and AbstractQueueBackend contracts and their tests, including context caching, batching delegation, and injected spill storage. Done means the package has provider-independent interfaces without a concrete SQS backend.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, azure, javascript
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.