aws / aws/aws-cdk

stepfunctions-tasks: Cannot use fromJsonPathAt when passing arg array values to ScriptBootstrapActionConfigProperty

Open
#29,299 7 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-stepfunctions-tasks bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

When writing step functions, often I want to pass a value from the payload to an API that expects an array value like this:
```ts
bootstrapActions: [
{
name: 'MyBootstrapAction',
scriptBootstrapAction: {
path: 'my_path',
"Args.$": '$.bootstrapArgs' // $.bootstrapArgs is an array value
},
},
],
```
or
```ts
bootstrapActions: [
{
name: 'MyBootstrapAction',
scriptBootstrapAction: {
path: 'my_path',
args: sfn.TaskInput.fromJsonPathAt('$.bootstrapArgs').value
},
},
],
```

This works for EmrAddStepsProps `args` property, but not for the args property within `ScriptBootstrapActionConfigProperty`. This means I cannot pass custom arguments based on my payload to bootstrap arguments when creating clusters, and this behavior seems contradictory to other tasks I have used in the library.

### Expected Behavior

I expected that I could dynamically pass a payload value to the `args` property within `ScriptBootstrapActionConfigProperty` inside of `EmrCreateClusterProps`. This should correspond to the task in the Step Functions console having an "Args.$" key instead of "Args". However, I cannot do this via CDK. I can do this via manual cloudformation and editing in the console, so this seems to be a CDK limitation.

### Current Behavior

Instead, it fails:
````bash
TypeError: x.map is not a function

at /Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/core/lib/runtime.js:1:1549

at BootstrapActionConfigToJson (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions-tasks/lib/emr/private/cluster-utils.js:1:10250)

at Array.map ()

at /Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/core/lib/runtime.js:1:1549

at EmrCreateCluster._renderTask (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.js:1:3931)

at EmrCreateCluster.toStateJson (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/states/task-base.js:1:2233)

at StateGraph.toGraphJson (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/state-graph.js:1:1984)

at ChainDefinitionBody.bind (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/state-machine.js:1:11644)

at new StateMachine (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/state-machine.js:1:6100)

at PersistentClusterStateMachine.createPersistentClusterStateMachine (/Volumes/workplace/MyWorkplace/src/MyService/dist/lib/constructs/stepfunctions/persistentClusterStateMachine.js:47:16)
````

### Reproduction Steps

In CDK, I have done this before with the following design pattern successfully:

```ts
const myTask = new sfn.Pass(this, 'FormatArgs', {
parameters: {
'emrStepArgs.$': `States.Array(
'spark-submit',
'--deploy-mode', 'cluster',
'--class','${this.customValue}',
$.payload.customValueFromPayload
)',
clusterId: sfn.TaskInput.fromJsonPathAt('$.clusterId').value,
job: sfn.TaskInput.fromJsonPathAt('$.job').value
}
}).next(
new tasks.EmrAddStep(this, 'RunStep', {
name: 'TaskName',
clusterId: sfn.TaskInput.fromJsonPathAt('$.clusterId').value,
jar: 'command-runner.jar',
args: sfn.TaskInput.fromJsonPathAt('$.emrStepArgs').value
})
```

If you notice, in `EmrAddStepProps`, args is expected to be an array, but allows this as is receiving this array from the input of the payload.

However, this does not work for the EmrCreateCluster api, specifically for the `args` property in `ScriptBootstrapActionConfigProperty`. Here is an example that fails to build:

````ts
const myTask = new sfn.Pass(this, 'FormatBootstrapArgs, {
{
parameters: {
'payload.$': '$.payload',
'bootstrapArgs.$': `States.Array(
'string_literal',
'${this.customValue}',
$.payload.customValueFromPayload
)',
}
}).next(
new tasks.EmrCreateCluster(this, 'CreateEmrCluster', {
name: 'ClusterName',
releaseLabel: '7.0.0',
instances: ...
clusterRole: ...
serviceRole: ...
applications: ....
configuration: ...
bootstrapActions: [
{
name: 'MyBootstrapAction',
scriptBootstrapAction: {
path: 'my_path',
args: sfn.TaskInput.fromJsonPathAt('$.bootstrapArgs').value
},
},
],
....
});
````

I can actually bypass this error when doing:
````ts
bootstrapActions: sfn.TaskInput.fromObject([
{
name: 'MyBootstrapAction',
scriptBootstrapAction: {
path: 'my_path',
'Args.$': '$.bootstrapArgs'
},
},
]),
````

But then the Args parameter is dropped on synthesis and it doesn't get published to the AWS account

### Possible Solution
It seems that sfn.TaskInput.fromJsonPathAt creates a resolvable token that is a string under the hood. However, CDK is still trying to map over this token which is incorrect behavior. I think that a possible fix could be to ignore this mapping behavior on resolvable tokens.

_No response_

### Additional Information/Context

_No response_

### CDK CLI Version

2.126.0

### Framework Version

_No response_

### Node.js Version

v18.19.1

### OS

Sonoma 14.3.1

### Language

TypeScript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in aws-stepfunctions-tasks/lib/emr/private/cluster-utils.js, following BootstrapActionConfigToJson, and compare its handling with the working EmrAddStep args path. Check the synthesized output from EmrCreateCluster and confirm that a TaskInput JSON path produces an Args.$ entry containing the array value without the x.map error.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.