[Feature Request]: Writing Parameters via ETL Metadata
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
It would be beneficial to have an option in the ETL metadata to pass variables and parameters to the pipeline model.
Regarding parameters, ideally, the default value of each parameter in the pipeline should be defined. This would facilitate debugging. Often, values for steps like lookup and insert are variable fields passed from a pipeline to a subpipeline via parameter and then passed to the pipeline model via ETL metadata. However, when testing the pipeline model generated by the ETL metadata, it is necessary to specify this variable repeatedly, which slows down the process. If possible, setting the parameter value and defining variables directly would simplify the process.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
Contributor guide
Research direction
Start by reviewing how ETL metadata creates the pipeline model and how pipeline and subpipeline parameters are currently passed. Define the expected behavior for metadata-supplied variables, parameters, and parameter defaults, including how those values are used when testing the generated model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100