kedro-org / kedro-org/kedro-devrel

Create one or more blog posts about Kedro & "dynamic pipelines" that sets out the various requirements, solutions and links to docs

Open
#7 13 comments 0 reactions 0 assignees View on GitHub
Blog post creation
Dominant language
No language data
Stars
0
Forks
4
PR merge metrics
No merged PRs in 30d

Description

It would be great if we had a page explaining Kedro and dynamic pipelines and what the best solutions are if you can't use Kedro.

We are asked about it regularly:

> Hello, good afternoon, I was adding a catalog item from a database that has timeseries data and I wanted to have a dynamic param so when I run the code It retrieves the data regarding a certain day.
I ran into this post:
https://github.com/kedro-org/kedro/issues/1089
But I am failing to understand what is the reasoning behind. I would like to understand what would be the approach in kedro. Loading the whole table from sql and then run the transformations in code. But that would make quite inefficient when running in production. (edited)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked Kedro issue #1089 and the customer question in this issue. Research the requirements and possible solutions for dynamic pipelines, including the SQL time-series example and alternatives when Kedro cannot be used. Done means one or more blog posts clearly explain the problem, solutions, trade-offs, and relevant documentation links.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
content, data, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.