apache / apache/hop

[Feature Request]: Metadata structure of stream - performance improvements

Open
#4,800 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage P3 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### What would you like to happen?

It seems that the **Metadata structure of stream** transform needs to process all the incoming rows to get the output. However, unless you actually need the total number of rows (this value is disabled by default), you just need a single row in the stream to get the actual metadata structure.

My suggestion is implementing a row sampling in the transform, and activate it whether the `Output row count?` flag is disabled or not.

The current workaround consists in adding a **Sample rows** transform before it, and set it to forward only the first row (it indeed works, especially if the incoming stream contains thousands of rows...)

### Issue Priority

Priority: 3

### Issue Component

Component: Transforms

Contributor guide

Open the contributing guide

Research direction

Start by locating the Metadata structure of stream transform and compare its processing with the Sample rows transform workaround. Check how the Output row count? flag affects processing; done means the transform can sample the needed row while preserving correct metadata and row-count behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.