stepfunctions-tasks: Cannot use fromJsonPathAt when passing arg array values to ScriptBootstrapActionConfigProperty
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
When writing step functions, often I want to pass a value from the payload to an API that expects an array value like this:
```ts
bootstrapActions: [
{
name: 'MyBootstrapAction',
scriptBootstrapAction: {
path: 'my_path',
"Args.$": '$.bootstrapArgs' // $.bootstrapArgs is an array value
},
},
],
```
or
```ts
bootstrapActions: [
{
name: 'MyBootstrapAction',
scriptBootstrapAction: {
path: 'my_path',
args: sfn.TaskInput.fromJsonPathAt('$.bootstrapArgs').value
},
},
],
```
This works for EmrAddStepsProps `args` property, but not for the args property within `ScriptBootstrapActionConfigProperty`. This means I cannot pass custom arguments based on my payload to bootstrap arguments when creating clusters, and this behavior seems contradictory to other tasks I have used in the library.
### Expected Behavior
I expected that I could dynamically pass a payload value to the `args` property within `ScriptBootstrapActionConfigProperty` inside of `EmrCreateClusterProps`. This should correspond to the task in the Step Functions console having an "Args.$" key instead of "Args". However, I cannot do this via CDK. I can do this via manual cloudformation and editing in the console, so this seems to be a CDK limitation.
### Current Behavior
Instead, it fails:
````bash
TypeError: x.map is not a function
at /Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/core/lib/runtime.js:1:1549
at BootstrapActionConfigToJson (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions-tasks/lib/emr/private/cluster-utils.js:1:10250)
at Array.map ()
at /Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/core/lib/runtime.js:1:1549
at EmrCreateCluster._renderTask (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions-tasks/lib/emr/emr-create-cluster.js:1:3931)
at EmrCreateCluster.toStateJson (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/states/task-base.js:1:2233)
at StateGraph.toGraphJson (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/state-graph.js:1:1984)
at ChainDefinitionBody.bind (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/state-machine.js:1:11644)
at new StateMachine (/Volumes/workplace/MyWorkplace/src/MyService/node_modules/aws-cdk-lib/aws-stepfunctions/lib/state-machine.js:1:6100)
at PersistentClusterStateMachine.createPersistentClusterStateMachine (/Volumes/workplace/MyWorkplace/src/MyService/dist/lib/constructs/stepfunctions/persistentClusterStateMachine.js:47:16)
````
### Reproduction Steps
In CDK, I have done this before with the following design pattern successfully:
```ts
const myTask = new sfn.Pass(this, 'FormatArgs', {
parameters: {
'emrStepArgs.$': `States.Array(
'spark-submit',
'--deploy-mode', 'cluster',
'--class','${this.customValue}',
$.payload.customValueFromPayload
)',
clusterId: sfn.TaskInput.fromJsonPathAt('$.clusterId').value,
job: sfn.TaskInput.fromJsonPathAt('$.job').value
}
}).next(
new tasks.EmrAddStep(this, 'RunStep', {
name: 'TaskName',
clusterId: sfn.TaskInput.fromJsonPathAt('$.clusterId').value,
jar: 'command-runner.jar',
args: sfn.TaskInput.fromJsonPathAt('$.emrStepArgs').value
})
```
If you notice, in `EmrAddStepProps`, args is expected to be an array, but allows this as is receiving this array from the input of the payload.
However, this does not work for the EmrCreateCluster api, specifically for the `args` property in `ScriptBootstrapActionConfigProperty`. Here is an example that fails to build:
````ts
const myTask = new sfn.Pass(this, 'FormatBootstrapArgs, {
{
parameters: {
'payload.$': '$.payload',
'bootstrapArgs.$': `States.Array(
'string_literal',
'${this.customValue}',
$.payload.customValueFromPayload
)',
}
}).next(
new tasks.EmrCreateCluster(this, 'CreateEmrCluster', {
name: 'ClusterName',
releaseLabel: '7.0.0',
instances: ...
clusterRole: ...
serviceRole: ...
applications: ....
configuration: ...
bootstrapActions: [
{
name: 'MyBootstrapAction',
scriptBootstrapAction: {
path: 'my_path',
args: sfn.TaskInput.fromJsonPathAt('$.bootstrapArgs').value
},
},
],
....
});
````
I can actually bypass this error when doing:
````ts
bootstrapActions: sfn.TaskInput.fromObject([
{
name: 'MyBootstrapAction',
scriptBootstrapAction: {
path: 'my_path',
'Args.$': '$.bootstrapArgs'
},
},
]),
````
But then the Args parameter is dropped on synthesis and it doesn't get published to the AWS account
### Possible Solution
It seems that sfn.TaskInput.fromJsonPathAt creates a resolvable token that is a string under the hood. However, CDK is still trying to map over this token which is incorrect behavior. I think that a possible fix could be to ignore this mapping behavior on resolvable tokens.
_No response_
### Additional Information/Context
_No response_
### CDK CLI Version
2.126.0
### Framework Version
_No response_
### Node.js Version
v18.19.1
### OS
Sonoma 14.3.1
### Language
TypeScript
### Language Version
_No response_
### Other information
_No response_
Contributor guide
Research direction
Start in aws-stepfunctions-tasks/lib/emr/private/cluster-utils.js, following BootstrapActionConfigToJson, and compare its handling with the working EmrAddStep args path. Check the synthesized output from EmrCreateCluster and confirm that a TaskInput JSON path produces an Args.$ entry containing the array value without the x.map error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100