getsops / getsops/sops

Error decrypting key: AccessDeniedException with AWS KMS

Open
#1,073 1 comment 13 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/keyservice keyservice/awskms
Dominant language
Go
Stars
23.1k
Forks
1.1k
Avg merge
1d 11h
Merged PRs (30d)
9

Description

Hello,

I've encountered an issue/QoL deficit which I haven't been able to find among other issues.

When I encrypt something as follows:

sops --aws-profile <profile> --kms <key arn> -e --output-type yaml <secret file> > secrets.encrypted.yaml

As part of the output, the following is generated:

sops:
    kms:
        - arn: arn:aws:kms:xxx
          created_at: "2022-06-08T14:21:13Z"
          enc: xxx
          aws_profile: xxx

When running this with FluxCD, and service account has permissions with an annotation to a role, it fails with an error similar to:

Error decrypting key: AccessDeniedException: The ciphertext
      | refers to a customer master key that does not exist, does
      | not exist in this region, or you are not allowed to access.
      | 	status code: 400

despite permissions being otherwise in order.

It appears to me that sops attempts to use the aws_profile used during encryption as generated at the encryption time.
Simply removing this part of the generated encrypted file will allow sops running inside of FluxCD to decrypt objects successfully.

This is also verifiable on CLI. Given no configured profiles, attempting to decrypt a file which has a sops.kms configuration with a specified aws_profile will fail with a similar error as above.

The QoL part of the issue is that encrypting a file with an aws_profile will require further manual edits on the encrypted file to ensure it can be consumed by Flux, also decrypting the file on CLI with:

sops --aws-profile <profile> --kms <key id> -d secrets.encrypted.yaml will not work unless aws_profile is specified on the file as well, even though we are passing the --aws-profile on CLI.

In summary, one has to be very careful with encrypting/decrypting files with sops if they are using an aws profile and have to manually modify the files after the fact for FuxCD consumption. This issue has personally cost me hours of debugging before I realized why sops on Flux was failing to decrypt my secrets.

My suggestion would be to not assume that the entity encrypting and the entity decrypting have equal credential configuration. I think SOPS should drop the aws_profile parameter it bakes into encrypted files and instead use default credentials, default should still be overridable by .sops config file.

This means that if you encrypt a file using an aws profile, and attempt to decrypt it without using one, it should be down to your default credentials / .sops config
If you attempt to decrypt it while specifying --aws-profile then it should take that with highest priority.

Let me know if you need any further clarification.
Cheers!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing AWS KMS handling for the sops.kms.aws_profile metadata and the --aws-profile CLI option. Reproduce the shown encryption and decryption commands with and without configured profiles, then verify behavior through FluxCD-style credentials. Done means credential precedence is consistent and encrypted files do not require manual aws_profile removal.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cli, cloud, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.