aws / aws/aws-cdk

(step functions): Add support to discard output in distributed map state

Open
#24,799 4 comments 2 reactions 0 assignees View on GitHub
@aws-cdk/aws-lambda effort/medium feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

Currently, the distributed `Map` state does not support the discard of its output as the default `Map` state does.

The following example always generates the output payload, causing the whole execution to fail due to `States.DataLimitExceeded` exception even though the [`OutputPath`](https://docs.aws.amazon.com/step-functions/latest/dg/input-output-outputpath.html) property is set to `null`:

```
new CustomState(this, 'ExampleMapStateS3Fetcher', {
stateJson: {
Type: 'Map',
MaxConcurrency: 1,
ItemReader: {
Resource: 'arn:aws:states:::s3:getObject',
Parameters: {
'Bucket.$': JsonPath.stringAt('$.bucketName'),
'Key.$': JsonPath.stringAt('$.objectName'),
},
ReaderConfig: {
InputType: 'JSON',
},
},
ItemProcessor: {
...(this.buildItemProcessingMap().toStateJson() as any).Iterator,
ProcessorConfig: {
Mode: 'DISTRIBUTED',
ExecutionType: 'STANDARD',
},
},
ResultPath: null, // ! -- no effect -- !
OutputPath: null, // ! -- no effect -- !
},
});
```

This results in the following exception message whenever the output payload is significantly large (order of thousands of iterations):

`The state/task '…' returned a result with a size exceeding the maximum number of bytes service limit.`

### Use Case

This feature is important whenever there is no need to use the output of the distributed Map's execution (discard output).

Currently, the workaround in order to not exceed the 256KB payload limit is to dump the data to an S3 bucket (through [`ResultWriter`](https://docs.aws.amazon.com/step-functions/latest/dg/input-output-resultwriter.html)):

```
ResultWriter: {
Resource: 'arn:aws:states:::s3:putObject',
Parameters: {
Bucket: 'bucket-name',
Prefix: 'results',
},
}
```

This however is unnecessary for cases where there is no need to inspect the state's output and thus causing harder integration effort (creation of the S3 bucket and appropriate IAM permissions) as well as incurring more costs due to the uploaded data into S3.

### Proposed Solution

Add support to discard state's output: accept nullable `OutputPath`/`ResultWriter` property.

### Other Information

_No response_

### Acknowledgements

- [X] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.55.0

### Environment details (OS name and version, etc.)

MacOS Ventura 13.2.1

Contributor guide

Open the contributing guide

Research direction

Start by tracing the distributed Map state implementation and the state JSON/type handling for OutputPath and ResultWriter. Compare this with the default Map state behavior and the documented ResultWriter workaround; done means a distributed Map can discard its output without producing a payload that exceeds the service limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.