(step functions): Add support to discard output in distributed map state
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the feature
Currently, the distributed `Map` state does not support the discard of its output as the default `Map` state does.
The following example always generates the output payload, causing the whole execution to fail due to `States.DataLimitExceeded` exception even though the [`OutputPath`](https://docs.aws.amazon.com/step-functions/latest/dg/input-output-outputpath.html) property is set to `null`:
```
new CustomState(this, 'ExampleMapStateS3Fetcher', {
stateJson: {
Type: 'Map',
MaxConcurrency: 1,
ItemReader: {
Resource: 'arn:aws:states:::s3:getObject',
Parameters: {
'Bucket.$': JsonPath.stringAt('$.bucketName'),
'Key.$': JsonPath.stringAt('$.objectName'),
},
ReaderConfig: {
InputType: 'JSON',
},
},
ItemProcessor: {
...(this.buildItemProcessingMap().toStateJson() as any).Iterator,
ProcessorConfig: {
Mode: 'DISTRIBUTED',
ExecutionType: 'STANDARD',
},
},
ResultPath: null, // ! -- no effect -- !
OutputPath: null, // ! -- no effect -- !
},
});
```
This results in the following exception message whenever the output payload is significantly large (order of thousands of iterations):
`The state/task '…' returned a result with a size exceeding the maximum number of bytes service limit.`
### Use Case
This feature is important whenever there is no need to use the output of the distributed Map's execution (discard output).
Currently, the workaround in order to not exceed the 256KB payload limit is to dump the data to an S3 bucket (through [`ResultWriter`](https://docs.aws.amazon.com/step-functions/latest/dg/input-output-resultwriter.html)):
```
ResultWriter: {
Resource: 'arn:aws:states:::s3:putObject',
Parameters: {
Bucket: 'bucket-name',
Prefix: 'results',
},
}
```
This however is unnecessary for cases where there is no need to inspect the state's output and thus causing harder integration effort (creation of the S3 bucket and appropriate IAM permissions) as well as incurring more costs due to the uploaded data into S3.
### Proposed Solution
Add support to discard state's output: accept nullable `OutputPath`/`ResultWriter` property.
### Other Information
_No response_
### Acknowledgements
- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.55.0
### Environment details (OS name and version, etc.)
MacOS Ventura 13.2.1
Contributor guide
Research direction
Start by tracing the distributed Map state implementation and the state JSON/type handling for OutputPath and ResultWriter. Compare this with the default Map state behavior and the documented ResultWriter workaround; done means a distributed Map can discard its output without producing a payload that exceeds the service limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100