awslabs / awslabs/aws-solutions-constructs

SageMaker endpoints deployments only allow container images from the current account

Open
#986 0 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
TypeScript
Stars
1.4k
Forks
268
Avg merge
5h 18m
Merged PRs (30d)
5

Description

When trying to deploy a SageMaker endpoint using aws-solutions-constructs, such as LambdaToSagemakerEndpoint, if the inference docker container image provided is hosted in a different account than the current one, it fails because of missing IAM permissions.
This prevents to use any of the [Pre-built SageMaker Docker images](https://docs.aws.amazon.com/sagemaker/latest/dg/docker-containers-prebuilt.html) that are hosted on various AWS-owned accounts depending on the regions.
See [Amazon SageMaker ECR Paths](https://docs.aws.amazon.com/sagemaker/latest/dg-ecr-paths/sagemaker-algo-docker-registry-paths.html)

### Reproduction Steps

const constructProps: LambdaToSagemakerEndpointProps = {
modelProps: {
primaryContainer: {
image: '763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.8-cpu',
modelDataUrl: 's3://jumpstart-cache-prod-eu-west-1/tensorflow-infer/infer-tensorflow-ic-swin-base-patch4-window7-224.tar.gz',
},
},
lambdaFunctionProps: {
runtime: lambda.Runtime.PYTHON_3_10,
handler: 'index.handler',
code: lambda.Code.fromInline(`
def handler(event, context):
print('Hello Lambda')
`)
},
};

const lambda_sagemaker = new LambdaToSagemakerEndpoint(this, 'LambdaToSagemakerEndpointPattern', constructProps);

### Error Log

9:52:55 AM | CREATE_FAILED | AWS::SageMaker::Endpoint | LambdaToSagemakerE.../SagemakerEndpoint
The role 'arn:aws:iam::_masked-account_:role/IcServerlessInferenceStac-LambdaToSagemakerEndpoin-B318Y1J1FRYS' does not have BatchGetImage permission for the image: '763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.8-cpu'.

### Environment

- **CDK CLI Version :** 2.92.0
- **CDK Framework Version:** 2
- **AWS Solutions Constructs Version :** 2.42.0
- **OS :** macOS Ventura 13.3
- **Language :** English

### Other

The permissions of the generated IAM role are restricted to the current account only.
This is what prevents the desired image to be retrieved from its ECR repository.

See [aws-solutions-constructs/core/lib/sagemaker-helper.ts#L128C1-L128C1](https://github.com/awslabs/aws-solutions-constructs/blob/555ff391854d17a70a32ffcaeb19975181f4c8ee/source/patterns/%40aws-solutions-constructs/core/lib/sagemaker-helper.ts#L128C1-L128C1)

```
// To create a Sagemaker model using Bring-Your-Own-Model (BYOM) algorith image
// The image URL is specified in the modelProps
role.addToPolicy(
new iam.PolicyStatement({
resources: [`arn:${cdk.Aws.PARTITION}:ecr:${cdk.Aws.REGION}:${cdk.Aws.ACCOUNT_ID}:repository/*`],
actions: [
'ecr:BatchCheckLayerAvailability',
'ecr:GetDownloadUrlForLayer',
'ecr:DescribeRepositories',
'ecr:DescribeImages',
'ecr:BatchGetImage',
],
})
);
```

---

This is :bug: Bug Report

Contributor guide

Open the contributing guide

Research direction

Start in source/patterns/@aws-solutions-constructs/core/lib/sagemaker-helper.ts around the referenced IAM policy, then trace how LambdaToSagemakerEndpoint creates the SageMaker model role. Verify the required ECR read permissions for an image hosted outside the current account and reproduce the provided cross-account deployment. Done means the endpoint can use the documented pre-built image without removing necessary permission restrictions.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.