[Feature Request]: Add "Pass Input Fields to Output" Option in Mapping/Pipeline Executor Transforms
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 19h
- Merged PRs (30d)
- 227
Description
### What would you like to happen?
Description: Issue Type: Improvement / Feature Request Priority: Major
Summary: Currently, when using a Mapping (Simple Mapping) or Pipeline Executor transform, fields from the main pipeline's input stream are not automatically propagated to the output step unless they are explicitly defined in the "Mapping Input" within the sub-pipeline and mapped back out.
If a developer simply wants to "enrich" a stream using a sub-pipeline (mapping) but keep all original fields (IDs, metadata, etc.), they are forced to manually map every single field inside the sub-pipeline configuration. This is time-consuming and prone to errors if the upstream data structure changes.
Proposed Solution: Add a checkbox or toggle option in the configuration dialog (e.g., "Main" or "Parameters" tab) labeled: "Pass all input fields to output" or "Propagate parent fields".
Behavior:
Enabled: All fields entering the Mapping transform are automatically available in the output stream, merged with any new fields created by the mapping.
Disabled: (Current behavior) Only fields explicitly returned by the Mapping Output are passed to the next step.
Benefit: This would significantly reduce development time for reusable components where the sub-pipeline performs a specific calculation or lookup but needs to preserve the original row context (IDs, timestamps) without manual mapping overhead.
### Issue Priority
Priority: 3
### Issue Component
Component: Pipelines
Contributor guide
Research direction
Start by locating the Mapping (Simple Mapping) and Pipeline Executor transform configuration dialogs, especially their Main or Parameters tabs, then trace how input and output fields are handled. Review existing transform tests if present; done means an enabled option preserves parent input fields alongside mapped fields, while disabled behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100