elastic / elastic/elastic-serverless-forwarder
Allow SSE-KMS (customer managed key) on the continuing and replay queues
- Dominant language
- Python
- Stars
- 38
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the enhancement:**
The four queues the templates create — continuing queue, continuing DLQ, replay queue, replay DLQ — hardcode `SqsManagedSseEnabled: true` with no parameter to override it (`.internal/aws/cloudformation/application.yaml`, lines 44, 52, 59 and 67):
```yaml
ElasticServerlessForwarderContinuingQueue:
Type: AWS::SQS::Queue
Properties:
...
SqsManagedSseEnabled: true
```
This is consistent with the original intent in #305, which proposed exactly this:
> my suggestion is to make the two queues encrypted by default, without the option of opting-out from encryption. We'll generate the KMS Keys at deployment time and configure the proper settings for the two queues.
The outcome in #353 was SSE-SQS instead, and the reason recorded on #305 was performance parity rather than any objection to SSE-KMS:
> Current key reuse period for Managed SSE for SQS is set to 24 hours (86400 seconds), so the performance impact for an SQS queue with enabled Managed SSE (SSE-SQS) should be roughly the same as with customer-managed SSE (SSE-KMS).
The stated goal of #353 was to force SSE on stacks created before Sept/Oct 2022. An opt-in parameter preserves that goal completely, since `SqsManagedSseEnabled: true` stays the default.
**Two changes are needed, not one.**
*1. A parameter for the queue key*, e.g. `ElasticServerlessForwarderInternalQueuesKMSKey`. When empty, behaviour is unchanged. When set, the four queues get `KmsMasterKeyId` instead of `SqsManagedSseEnabled`.
*2. The generated IAM policy needs `kms:GenerateDataKey`.* Without this the parameter alone is not usable. The macro grants the Lambda `sqs:SendMessage` on both queues (`.internal/aws/cloudformation/macro.yaml`, line 251), but the only KMS action it ever grants is `kms:Decrypt` (line 216):
```python
if "ElasticServerlessForwarderKMSKeys" in parameters:
kms_keys_arn = [x for x in parameters["ElasticServerlessForwarderKMSKeys"] if len(x.strip()) > 0]
if len(kms_keys_arn) > 0:
policy_fragment["Properties"]["PolicyDocument"]["Statement"].append(
{
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": kms_keys_arn
}
)
```
Writing to an SSE-KMS queue requires `kms:GenerateDataKey`. Rather than widening the existing statement — which covers input sources and should stay read-only — a separate statement scoped to the internal queue key would be cleaner:
```python
{
"Effect": "Allow",
"Action": ["kms:Decrypt", "kms:GenerateDataKey"],
"Resource": internal_queues_kms_key_arn
}
```
The failure mode without this is quiet: ordinary batches keep forwarding while continuation for large files and replay for failed batches fail, because reading needs only `Decrypt` (which is granted) and writing needs `GenerateDataKey` (which is not).
As an alternative, generating the CMK in the template — as #305 originally proposed — would avoid a parameter entirely. Either approach resolves the control.
**Describe a specific use case for the enhancement or feature:**
We deploy ESF across many AWS accounts under a compliance regime that requires a customer managed key for data at rest. Every queue we create ourselves can be moved to SSE-KMS, but these four cannot, and they carry log content so they are in scope for the control.
Our current workaround is to call `SetQueueAttributes` on the four queues out of band after the stack deploys. It is fragile in three ways: the queue names embed the nested stack UUID so they have to be discovered at runtime, the change is invisible to the IaC that deployed the stack and to `terraform plan`, and it requires separately attaching `kms:GenerateDataKey` to the Lambda role because of the gap described above. A supported parameter would remove all three problems.
Contributor guide
Research direction
Start with .internal/aws/cloudformation/application.yaml at the four queue definitions and .internal/aws/cloudformation/macro.yaml around the existing KMS policy generation. Trace how parameters reach the templates and generated IAM policy. Done means the default remains SSE-SQS, an optional customer-managed key configures all four queues, and the Lambda role can write to them with the required KMS permission.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, yaml
- Domain
- cloud, infrastructure, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100