aws / aws/aws-cdk

(step_functions): S3 Json Lines Item Reader

Open
#33,601 6 comments 1 reaction 0 assignees View on GitHub
@aws-cdk/aws-stepfunctions effort/medium feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the feature

The AWS Step Functions team recently introduced support for JSON Lines (JSONL) in Distributed Map, allowing efficient processing of large datasets stored in this format:
🔗 [AWS Blog Post – JSONL Support in Step Functions Distributed Map](https://aws.amazon.com/blogs/compute/introducing-jsonl-support-with-step-functions-distributed-map/)

Currently, the AWS CDK provides S3JsonItemReader ([docs](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_stepfunctions/S3JsonItemReader.html)), which supports reading JSON objects from an S3 file. However, this construct does not support JSONL files. Given that JSONL is now natively supported by Step Functions Distributed Map, it would be highly beneficial to have native support for JSONL in the CDK as well.

### Use Case

Developers using AWS Step Functions with CDK would be able to seamlessly leverage JSONL for large-scale data processing, without resorting to custom implementations or workarounds.

### Proposed Solution

Introduce a new construct (or extend the existing S3JsonItemReader) to support JSONL files, aligning with the latest Step Functions capabilities.

Example:

```python
s3_jsonl_reader = stepfunctions.S3JsonItemReader(
bucket=s3_bucket,
key="data.jsonl",
format=stepfunctions.JsonFormat.JSONL # Example of possible new parameter
)
```

### Other Information

_No response_

### Acknowledgements

- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CDK version used

2.280.0

### Environment details (OS name and version, etc.)

Windows 11, python

Contributor guide

Open the contributing guide

Research direction

Start with the existing S3JsonItemReader construct and compare its behavior with the AWS Step Functions JSONL support described in the linked blog post. Decide whether to extend the construct or add a new one, then verify that the CDK exposes JSONL configuration for Distributed Map without breaking existing JSON readers.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.