elastic / elastic/elastic-serverless-forwarder

Allow SSE-KMS (customer managed key) on the continuing and replay queues

Open
#1,058 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
38
Forks
50
PR merge metrics
No merged PRs in 30d

Description

**Describe the enhancement:**

The four queues the templates create — continuing queue, continuing DLQ, replay queue, replay DLQ — hardcode `SqsManagedSseEnabled: true` with no parameter to override it (`.internal/aws/cloudformation/application.yaml`, lines 44, 52, 59 and 67):

```yaml
ElasticServerlessForwarderContinuingQueue:
Type: AWS::SQS::Queue
Properties:
...
SqsManagedSseEnabled: true
```

This is consistent with the original intent in #305, which proposed exactly this:

> my suggestion is to make the two queues encrypted by default, without the option of opting-out from encryption. We'll generate the KMS Keys at deployment time and configure the proper settings for the two queues.

The outcome in #353 was SSE-SQS instead, and the reason recorded on #305 was performance parity rather than any objection to SSE-KMS:

> Current key reuse period for Managed SSE for SQS is set to 24 hours (86400 seconds), so the performance impact for an SQS queue with enabled Managed SSE (SSE-SQS) should be roughly the same as with customer-managed SSE (SSE-KMS).

The stated goal of #353 was to force SSE on stacks created before Sept/Oct 2022. An opt-in parameter preserves that goal completely, since `SqsManagedSseEnabled: true` stays the default.

**Two changes are needed, not one.**

*1. A parameter for the queue key*, e.g. `ElasticServerlessForwarderInternalQueuesKMSKey`. When empty, behaviour is unchanged. When set, the four queues get `KmsMasterKeyId` instead of `SqsManagedSseEnabled`.

*2. The generated IAM policy needs `kms:GenerateDataKey`.* Without this the parameter alone is not usable. The macro grants the Lambda `sqs:SendMessage` on both queues (`.internal/aws/cloudformation/macro.yaml`, line 251), but the only KMS action it ever grants is `kms:Decrypt` (line 216):

```python
if "ElasticServerlessForwarderKMSKeys" in parameters:
kms_keys_arn = [x for x in parameters["ElasticServerlessForwarderKMSKeys"] if len(x.strip()) > 0]
if len(kms_keys_arn) > 0:
policy_fragment["Properties"]["PolicyDocument"]["Statement"].append(
{
"Effect": "Allow",
"Action": "kms:Decrypt",
"Resource": kms_keys_arn
}
)
```

Writing to an SSE-KMS queue requires `kms:GenerateDataKey`. Rather than widening the existing statement — which covers input sources and should stay read-only — a separate statement scoped to the internal queue key would be cleaner:

```python
{
"Effect": "Allow",
"Action": ["kms:Decrypt", "kms:GenerateDataKey"],
"Resource": internal_queues_kms_key_arn
}
```

The failure mode without this is quiet: ordinary batches keep forwarding while continuation for large files and replay for failed batches fail, because reading needs only `Decrypt` (which is granted) and writing needs `GenerateDataKey` (which is not).

As an alternative, generating the CMK in the template — as #305 originally proposed — would avoid a parameter entirely. Either approach resolves the control.

**Describe a specific use case for the enhancement or feature:**

We deploy ESF across many AWS accounts under a compliance regime that requires a customer managed key for data at rest. Every queue we create ourselves can be moved to SSE-KMS, but these four cannot, and they carry log content so they are in scope for the control.

Our current workaround is to call `SetQueueAttributes` on the four queues out of band after the stack deploys. It is fragile in three ways: the queue names embed the nested stack UUID so they have to be discovered at runtime, the change is invisible to the IaC that deployed the stack and to `terraform plan`, and it requires separately attaching `kms:GenerateDataKey` to the Lambda role because of the gap described above. A supported parameter would remove all three problems.

Contributor guide

Open the contributing guide

Research direction

Start with .internal/aws/cloudformation/application.yaml at the four queue definitions and .internal/aws/cloudformation/macro.yaml around the existing KMS policy generation. Trace how parameters reach the templates and generated IAM policy. Done means the default remains SSE-SQS, an optional customer-managed key configures all four queues, and the Lambda role can write to them with the required KMS permission.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, yaml
Domain
cloud, infrastructure, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.