[Feature Request]: Build ETL/ELT alternative
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
Hello
We see in new tech like Datastage Next gen or other a simple way to switch the workload from ETL to ELT
In Datastage as example , there is a check box in every pipeline that says execute as ETL (default) and you can change it as execute as ELT
if all component in the pipeline are compatible , ( sorts, filter, join , etc..) you can switch to ELT , in that case instead of executing the job in the engine, it will generate and execute a dbt code
we can use dbt , or just generate sql or whatever easier comes to your mind but i think that would be the golden feature to have
I'm just suggesting :)
### Issue Priority
Priority: 3
### Issue Component
Component: Pipelines
Contributor guide
Research direction
No files or tests are named. Start by reviewing the Pipelines component and how pipeline steps such as sorts, filters, and joins are classified for compatibility; done would require an agreed design for switching eligible pipelines from engine execution to generated and executed SQL or dbt code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100