Granting S3 access to karpenter nodes
- Dominant language
- Shell
- Stars
- 857
- Forks
- 303
- Avg merge
- 11h 5m
- Merged PRs (30d)
- 3
Description
- [x] ✋ I have searched the open/closed issues and my issue is not listed.
#### Please describe your question here
Thanks for the great examples! I altered the jupyterhub on eks example (for a private cluster accessed via a Tailscale VPN) and I'm now adding a ray cluster and trying to grant S3 access to the jobs running on karpenter nodes. I was trying to use the same karpenter provisioners but how do I grant the jobs S3 access?
- The ray example [uses](https://github.com/awslabs/data-on-eks/blob/44cb0769afc752e57bfb2d11192ebcec1ce97389/ai-ml/ray/terraform/main.tf#L193C12-L193C63) the `terraform-aws-modules/eks/aws//modules/karpenter` module and attaches the relevant policies via the `iam_role_additional_policies` argument which is pretty straightforward.
- The jupyterhub example (which I currently have running) [uses](https://github.com/awslabs/data-on-eks/blob/44cb0769afc752e57bfb2d11192ebcec1ce97389/ai-ml/jupyterhub/addons.tf#L34C14-L34C46) `aws-ia/eks-blueprints-addons/aws` which ultimately [uses](https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/d45aada0bf9bc87589d3dc984c0b917599eace78/main.tf#L2930) `aws-ia/eks-blueprints-addon/aws`. The two things I've tried that hasn't worked is:
1. attaching the relevant policies via the `role_policies` [input](https://registry.terraform.io/modules/aws-ia/eks-blueprints-addon/aws/latest?tab=inputs)
```
karpenter = {
role_policies = {
bucket1_get_policy = bucket1_get_policy_arn
bucket2_get_policy = bucket2_get_policy_arn
}
}
```
2. using `aws_iam_role_policy_attachment` resources with `role = module.eks_blueprints_addons.karpenter.iam_role_name`.
```
resource "aws_iam_role_policy_attachment" "karpenter_s3_access" {
for_each = toset([
bucket1_get_policy_arn,
bucket2_get_policy_arn,
])
role = module.eks_blueprints_addons.karpenter.node_instance_profile_name
policy_arn = each.value
}
```
Also is there a preference for which module to use?
#### Provide a link to the example/module related to the question
[jupyterhub](https://github.com/awslabs/data-on-eks/blob/main/ai-ml/jupyterhub/addons.tf)
[ray](https://github.com/awslabs/data-on-eks/blob/main/ai-ml/ray/terraform/main.tf)
#### Additional context
I may just follow the ray example and generate karpenter resources outside of `aws-ia/eks-blueprints-addons/aws`.
Also here's the policy I'm attaching:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Download",
"Effect": "Allow",
"Action": [
"s3:List*",
"s3:Get*"
],
"Resource": [
"${bucket_arn}",
"${bucket_arn}/*"
]
},
{
"Sid": "Decrypt",
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": [
"${kms_key_arn}"
]
}
]
}
```
And the error I'm getting is:
```
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
```
Contributor guide
Assessment
This issue has not been assessed yet.