kedro-org / kedro-org/kedro-devrel
Create one or more blog posts about Kedro & "dynamic pipelines" that sets out the various requirements, solutions and links to docs
- Dominant language
- No language data
- Stars
- 0
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
It would be great if we had a page explaining Kedro and dynamic pipelines and what the best solutions are if you can't use Kedro.
We are asked about it regularly:
> Hello, good afternoon, I was adding a catalog item from a database that has timeseries data and I wanted to have a dynamic param so when I run the code It retrieves the data regarding a certain day.
I ran into this post:
https://github.com/kedro-org/kedro/issues/1089
But I am failing to understand what is the reasoning behind. I would like to understand what would be the approach in kedro. Loading the whole table from sql and then run the transformations in code. But that would make quite inefficient when running in production. (edited)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked Kedro issue #1089 and the customer question in this issue. Research the requirements and possible solutions for dynamic pipelines, including the SQL time-series example and alternatives when Kedro cannot be used. Done means one or more blog posts clearly explain the problem, solutions, trade-offs, and relevant documentation links.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- content, data, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100