aws / aws/aws-cdk

Add Event Target for Amazon SageMaker Model Building Pipelines

Open
#14,887 2 comments 2 reactions 0 assignees View on GitHub
@aws-cdk/aws-events-targets @aws-cdk/aws-sagemaker effort/large feature-request feature/coverage-gap p1
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
71

Description

Update the CDK [Event Targets for Amazon EventBridge](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-events-targets-readme.html) to support support [Amazon SageMaker Model Building Pipelines](https://docs.aws.amazon.com/sagemaker/latest/dg/pipeline-eventbridge.html) as a target.

SageMaker Pipeline [definitions](https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_sagemaker/CfnPipeline.html) are already supported in the CDK

### Use Case

There are number of scenarios when triggering a SageMaker pipeline would be useful.
* Scheduled Retraining
* Retraining on Drift Detection
* Batch Inference on new data arriving in S3.

### Proposed Solution
Create a new **SageMakerPipeline** event target, which will be similar to [StepFunctions](https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-events-targets/lib/state-machine.ts).

Would need to support list of `Name`:`Value` [PipelineParameters](https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_Target.html#eventbridge-Type-Target-SageMakerPipelineParameters)

Following is an example adapted from the [sate machine](https://docs.aws.amazon.com/cdk/api/latest/docs/aws-events-targets-readme.html#start-a-stepfunctions-state-machine)

```
import * as iam from '@aws-sdk/aws-iam';
import * as sm from '@aws-cdk/aws-sagemaker';
import * as targets from "@aws-cdk/aws-events-targets";

const rule = new events.Rule(stack, 'Rule', {
schedule: events.Schedule.rate(cdk.Duration.minutes(1)),
});

const role = new iam.Role(stack, 'Role', {
assumedBy: new iam.ServicePrincipal('events.amazonaws.com'),
});
const pipeline = new sm.CfnPipeline(stack, 'Pipeline', {
pipelineName: "TestPipeline",
roleArn: role,
pipelineDefinition:{
"PipelineDefinitionBody": "{\"Version\":\"2020-12-01\",\"Parameters\":[{\"Name\":\"InputDataSource\",\"DefaultValue\":\"\"},{\"Name\":\"InstanceCount\",\"Type\":\"Integer\",\"DefaultValue\":1}],\"Steps\":[{\"Name\":\"Training1\",\"Type\":\"Training\",\"Arguments\":{\"InputDataConfig\":[{\"DataSource\":{\"S3DataSource\":{\"S3Uri\":{\"Get\":\"Parameters.InputDataSource\"}}}}],\"OutputDataConfig\":{\"S3OutputPath\":\"s3://my-s3-bucket/\"},\"ResourceConfig\":{\"InstanceType\":\"ml.m5.large\",\"InstanceCount\":{\"Get\":\"Parameters.InstanceCount\"},\"VolumeSizeInGB\":1024}}}]}"
}
})

rule.addTarget(new targets.SageMakerPipeline(pipeline, {
input: events.RuleTargetInput.fromObject({ InstanceCount: '1' }),
}));
```

* [x] :wave: I may be able to implement this feature request
* [ ] :warning: This feature might incur a breaking change

---

This is a :rocket: Feature Request

Contributor guide

Open the contributing guide

Research direction

Start with packages/@aws-cdk/aws-events-targets/lib/state-machine.ts and the Event Targets README linked in the issue. Compare the proposed SageMakerPipeline target with the StepFunctions target and verify how EventBridge PipelineParameters are represented. Done means the target is available for a SageMaker CfnPipeline and accepts the requested Name/Value parameters.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.