[Feature Request]: allow streaming output to file in the Enhanced JSON Output transform
Open
awaiting triage
P2
Transforms
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
the Enhanced JSON Output currently fails (OutOfMemoryException) when a pipeline tries to write a file that is larger than what fits in the available memory.
The transform needs to support streaming data to disk to resolved this.
### Issue Priority
Priority: 2
### Issue Component
Component: Transforms
Contributor guide
Research direction
Locate the Enhanced JSON Output transform and reproduce the OutOfMemoryException with a file larger than available memory. Trace how the transform currently writes output and verify that large pipeline outputs can be written to disk without requiring the entire file in memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100