aws / aws/aws-cdk

stepfunctions-tasks: EmrCreateClusterProps clusterRole should be instance profile not IRole

Open
#8,080 13 comments 5 reactions 0 assignees View on GitHub
@aws-cdk/aws-stepfunctions-tasks bug effort/small p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

Seems that the API for AWS CDK with AWS Step Function to create an EMR cluster is wrong.

With an EMR service object, we can provide `RunJobFlowInput` to `runJobFlow`, that specified `JobFlowRole: instanceProfile.ref`, with `instanceProfile` as a profile with a single role:

```
// Example role, our role is constructed with more permissions
const myRole = new iam.Role(stack, 'my-role', {
assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
managedPolicies: [iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonElasticMapReduceforEC2Role')],
});

const instanceProfile = new iam.CfnInstanceProfile(stack, `emr-ec2-instance-profile`, {
roles: [myRole.roleName],
});
```

With `stepfunctionsTasks.EmrCreateCluster`, however, the required `EmrCreateClusterProps`, have renamed `JobFlowRole` into `clusterRole`, and [expect](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-stepfunctions-tasks.EmrCreateClusterProps.html) `clusterRole` to be of type `iam.IRole` instead of a reference to the previous `iam.CfnInstanceProfile`. Attempting to pass the IAM Role that the profile was created with (e.g. `myRole`), results in an `Invalid InstanceProfile` error when running Step Function.

This seems to indicate that despite trying to pass a `IRole`, what the API should be requesting is the reference to a `CfnInstanceProfile`.

### Reproduction Steps

```
// Example role, our role is constructed with more permissions
const myRole = new iam.Role(stack, 'my-role', {
assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
managedPolicies: [iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonElasticMapReduceforEC2Role')],
});

const instanceProfile = new iam.CfnInstanceProfile(stack, `emr-ec2-instance-profile`, {
roles: [myRole.roleName],
});

const startClusterTask = new steps.Task(stack, 'Create Cluster', {
task: new tasks.EmrCreateCluster({
name: 'myCluster',
visibleToAllUsers: true,
releaseLabel: 'emr-6.0.0',
instances: {
masterInstanceType: 'm5.xlarge',
slaveInstanceType: 'm5.xlarge',
instanceCount: 3,
},
clusterRole: myRole, // aka JowFlowRole or instanceProfile.ref
configurations: [
{
classification: 'spark-env',
properties: {
PYSPARK_PYTHON: '/usr/bin/python3',
},
},
],
applications: [
{
name: 'Spark',
},
],
}),
});

const stopClusterTask = new steps.Task(stack, 'Task', {
task: new tasks.EmrTerminateCluster({
clusterId: 'myCluster',
}),
});

const workflow = steps.Chain.start(startClusterTask).next(stopClusterTask);

const stateMachine = new steps.StateMachine(stack, 'myStateMachine', {
stateMachineType: steps.StateMachineType.STANDARD,
definition: workflow,
});

```

### Error Log

Attempting to run the step function will yield:

```
{
"error": "EMR.AmazonElasticMapReduceException",
"cause": "Invalid InstanceProfile: ${profile}. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: ${Request})"
}
```

### Environment

- **CLI Version :** aws-cli/1.16.227
- **Framework Version:** 1.37.0
- **OS :** MacOS Catalina 10.15.4
- **Language :** Typescript

### Other

The type of `clusterRole` in `EmrCreateClusterProps` should not be `IRole`, but a reference to the instance profile, as expected in the [EMR specification](https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html#EMR-RunJobFlow-request-JobFlowRole).

---

This is :bug: Bug Report

Contributor guide

Open the contributing guide

Research direction

Start with the stepfunctions-tasks EmrCreateClusterProps and EmrCreateCluster entry points, then compare clusterRole with EMR RunJobFlow's JobFlowRole and the instanceProfile.ref shown in the reproduction. Reproduce the Invalid InstanceProfile failure and verify that the generated request uses an instance-profile reference; the issue is done when the task accepts and sends the expected value.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.