awslabs / awslabs/aws-solutions-constructs
SageMaker endpoints deployments only allow container images from the current account
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 268
- Avg merge
- 5h 18m
- Merged PRs (30d)
- 5
Description
When trying to deploy a SageMaker endpoint using aws-solutions-constructs, such as LambdaToSagemakerEndpoint, if the inference docker container image provided is hosted in a different account than the current one, it fails because of missing IAM permissions.
This prevents to use any of the [Pre-built SageMaker Docker images](https://docs.aws.amazon.com/sagemaker/latest/dg/docker-containers-prebuilt.html) that are hosted on various AWS-owned accounts depending on the regions.
See [Amazon SageMaker ECR Paths](https://docs.aws.amazon.com/sagemaker/latest/dg-ecr-paths/sagemaker-algo-docker-registry-paths.html)
### Reproduction Steps
const constructProps: LambdaToSagemakerEndpointProps = {
modelProps: {
primaryContainer: {
image: '763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.8-cpu',
modelDataUrl: 's3://jumpstart-cache-prod-eu-west-1/tensorflow-infer/infer-tensorflow-ic-swin-base-patch4-window7-224.tar.gz',
},
},
lambdaFunctionProps: {
runtime: lambda.Runtime.PYTHON_3_10,
handler: 'index.handler',
code: lambda.Code.fromInline(`
def handler(event, context):
print('Hello Lambda')
`)
},
};
const lambda_sagemaker = new LambdaToSagemakerEndpoint(this, 'LambdaToSagemakerEndpointPattern', constructProps);
### Error Log
9:52:55 AM | CREATE_FAILED | AWS::SageMaker::Endpoint | LambdaToSagemakerE.../SagemakerEndpoint
The role 'arn:aws:iam::_masked-account_:role/IcServerlessInferenceStac-LambdaToSagemakerEndpoin-B318Y1J1FRYS' does not have BatchGetImage permission for the image: '763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.8-cpu'.
### Environment
- **CDK CLI Version :** 2.92.0
- **CDK Framework Version:** 2
- **AWS Solutions Constructs Version :** 2.42.0
- **OS :** macOS Ventura 13.3
- **Language :** English
### Other
The permissions of the generated IAM role are restricted to the current account only.
This is what prevents the desired image to be retrieved from its ECR repository.
See [aws-solutions-constructs/core/lib/sagemaker-helper.ts#L128C1-L128C1](https://github.com/awslabs/aws-solutions-constructs/blob/555ff391854d17a70a32ffcaeb19975181f4c8ee/source/patterns/%40aws-solutions-constructs/core/lib/sagemaker-helper.ts#L128C1-L128C1)
```
// To create a Sagemaker model using Bring-Your-Own-Model (BYOM) algorith image
// The image URL is specified in the modelProps
role.addToPolicy(
new iam.PolicyStatement({
resources: [`arn:${cdk.Aws.PARTITION}:ecr:${cdk.Aws.REGION}:${cdk.Aws.ACCOUNT_ID}:repository/*`],
actions: [
'ecr:BatchCheckLayerAvailability',
'ecr:GetDownloadUrlForLayer',
'ecr:DescribeRepositories',
'ecr:DescribeImages',
'ecr:BatchGetImage',
],
})
);
```
---
This is :bug: Bug Report
Contributor guide
Research direction
Start in source/patterns/@aws-solutions-constructs/core/lib/sagemaker-helper.ts around the referenced IAM policy, then trace how LambdaToSagemakerEndpoint creates the SageMaker model role. Verify the required ECR read permissions for an image hosted outside the current account and reproduce the provided cross-account deployment. Done means the endpoint can use the documented pre-built image without removing necessary permission restrictions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100