Allow buildkit to use the pod's credentials when running on k8s
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 10.3k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 50
Description
When submitting a job to a k8s pod running buildkit-rootless:
buildctl --addr 'kube-pod://buildkitd-0?namespace=buildkit&container=buildkitd' build ...
Our setup does not allow developers to push to ECRs, it must be done via a 'service' (e.g. buildkit in this case). However buildkit will always try to use the user's local credentials, even when the pod its running in has sufficient permissions (the same setup works for kaniko):
=> [auth] sharing credentials for xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com 0.0s
------
> exporting to image:
------
error: failed to solve: failed to push xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/repo:tag: unexpected status: 403 Forbidden
Docs seems to suggest that when using S3 for caching, it's able to use IAM instance profiles / env vars / config files:
AWS Authentication:
The simplest way is to use an IAM Instance profile. Others options are:
Any system using environment variables / config files supported by the AWS Go SDK. The configuration must be available for the buildkit daemon, not for the client.
Can something similar be done when pushing to ECR? Instead of a user's local credentials being 'shared'..
Similar issues: https://github.com/moby/buildkit/issues/3947, https://github.com/moby/buildkit/issues/3772
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the buildctl kube-pod invocation and the daemon-side authentication behavior described in the issue, comparing ECR pushes with the documented S3 AWS authentication options. Determine whether the buildkit daemon can use the pod's IAM credentials instead of sharing the client's credentials, and define completion as a successful ECR push using only the pod's permissions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- authentication, build-system, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100