Need streaming option for output
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 523
- Forks
- 75
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 24
Description
Summary of the new feature / enhancement
When executing a large configuration, all the objects are held in memory and then serialized to JSON once the operation is complete. This can use a huge amount of ram that would affect other system operations.
Proposed technical implementation details (optional)
Introduce a new --output-format JSON-streaming option. This option does not return a complete JSON object, but instead returns:
- each output of a resource as a separate JSONline
- any final
executionInformationtype object is at the end (maybe not even needed in this use case?)
May need to consider different output structures of get, set, test, and export. For example, in set do we still output beforeState and afterState? Do we need a simple protocol to indicate which part of the doc is being emitted before streaming the JSONlines so that the consumer doesn't have to try to deserialize every possible type of object?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing output handling for get, set, test, and export, then define how the JSON-streaming option represents each resource and any final executionInformation object; done requires an agreed protocol that avoids retaining the full result in memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100