[Feature Request]: Metadata structure of stream - performance improvements
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
It seems that the **Metadata structure of stream** transform needs to process all the incoming rows to get the output. However, unless you actually need the total number of rows (this value is disabled by default), you just need a single row in the stream to get the actual metadata structure.
My suggestion is implementing a row sampling in the transform, and activate it whether the `Output row count?` flag is disabled or not.
The current workaround consists in adding a **Sample rows** transform before it, and set it to forward only the first row (it indeed works, especially if the incoming stream contains thousands of rows...)
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
Contributor guide
Research direction
Start by locating the Metadata structure of stream transform and compare its processing with the Sample rows transform workaround. Check how the Output row count? flag affects processing; done means the transform can sample the needed row while preserving correct metadata and row-count behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100