(step_functions): S3 Json Lines Item Reader
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the feature
The AWS Step Functions team recently introduced support for JSON Lines (JSONL) in Distributed Map, allowing efficient processing of large datasets stored in this format:
🔗 [AWS Blog Post – JSONL Support in Step Functions Distributed Map](https://aws.amazon.com/blogs/compute/introducing-jsonl-support-with-step-functions-distributed-map/)
Currently, the AWS CDK provides S3JsonItemReader ([docs](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_stepfunctions/S3JsonItemReader.html)), which supports reading JSON objects from an S3 file. However, this construct does not support JSONL files. Given that JSONL is now natively supported by Step Functions Distributed Map, it would be highly beneficial to have native support for JSONL in the CDK as well.
### Use Case
Developers using AWS Step Functions with CDK would be able to seamlessly leverage JSONL for large-scale data processing, without resorting to custom implementations or workarounds.
### Proposed Solution
Introduce a new construct (or extend the existing S3JsonItemReader) to support JSONL files, aligning with the latest Step Functions capabilities.
Example:
```python
s3_jsonl_reader = stepfunctions.S3JsonItemReader(
bucket=s3_bucket,
key="data.jsonl",
format=stepfunctions.JsonFormat.JSONL # Example of possible new parameter
)
```
### Other Information
_No response_
### Acknowledgements
- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### CDK version used
2.280.0
### Environment details (OS name and version, etc.)
Windows 11, python
Contributor guide
Research direction
Start with the existing S3JsonItemReader construct and compare its behavior with the AWS Step Functions JSONL support described in the linked blog post. Decide whether to extend the construct or add a new one, then verify that the CDK exposes JSONL configuration for Distributed Map without breaking existing JSON readers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100