getsops / getsops/sops

Ideas for debugging a timeout?

Open
#675 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
23.1k
Forks
1.1k
Avg merge
1d 11h
Merged PRs (30d)
9

Description

I have an issue that is probably not the fault of sops, but I can only replicate with sops — and so if anyone has hit something similar, or has thoughts on how I could debug this further, I'd appreciate any insights.

I'm using sops on a GKE cluster set up with Workload Identity. The following command fails:

kubectl run -it    --image gcr.io/[...] \
                      --serviceaccount argo-service-account \
                      --namespace default \
                      --rm \
                      -- test-pod bash -c 'sops -d [...]'

...with...

If you don't see a command prompt, try pressing enter.
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  projects/[...]/locations/us-west2/keyRings/[...]: FAILED
    - | Error decrypting key: Post
      | https://cloudkms.googleapis.com/v1/projects/[...]:decrypt?alt=json&prettyPrint=false:
      | Get
      | http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform:
      | net/http: timeout awaiting response headers

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

But the following command succeeds:

kubectl run -it    --image gcr.io/[...] \
                      --serviceaccount argo-service-account \
                      --namespace default \
                      --rm \
                      -- test-pod bash -c 'sleep 5 sops -d [...]' # <- note `sleep 5`

...so adding a sleep 5 remedies the failure. I've replicated a few times to ensure this is the reason. I can't replicate the failures with any calls to gcloud kms encrypt; I only get this behavior with sops.

Potentially sops is "too fast" and issues a request before GKE has had the chance to set up permissions for the pod?

Besides adding sleep 5 to every instantiation, is there any way to allow for a longer timeout with sops? Or any other ideas for debugging?

Thank you!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by reproducing the two kubectl commands in the issue, comparing the immediate sops -d call with the version preceded by sleep 5, and trace the Workload Identity metadata-token request. Done would mean identifying the timeout cause or a concrete sops configuration or behavior change that removes the workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, kubernetes
Domain
authentication, cli, cloud, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.