kedro-org / kedro-org/kedro-plugins
Use `pl.sink_*` in `LazyPolarsDataset._save`
- Dominant language
- Python
- Stars
- 119
- Forks
- 136
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 5
Description
## Description
When passing a lazy DataFrame to `LazyPolarsDataset`, it is currently collected into an eager DataFrame before writing it using the appropriate `pl.write_*` function. This can be skipped by writing the lazy dataframe using `pl.sink_*`.
## Context
In some cases, it may be faster to collect the lazy DataFrame in streaming mode. Additionally, it is not always possible to collect the entire DataFrame (e.g., if the data is too large). Using `pl.sink_*`, the entire data set does not need to be loaded.
## Possible Implementation
In the `_save` function, the input DataFrame could first be coerced into a lazy DataFrame and then written to disk using `pl.sink_*`.
## Possible Alternatives
Contributor guide
Research direction
Start at the LazyPolarsDataset._save entry point and inspect how lazy DataFrames are currently collected and passed to pl.write_* functions. Compare the available pl.sink_* APIs for the supported output formats, and confirm that saving avoids loading the entire dataset while preserving existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100