kedro-org / kedro-org/kedro-plugins

Use `pl.sink_*` in `LazyPolarsDataset._save`

Open
#702 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
119
Forks
136
Avg merge
4d 10h
Merged PRs (30d)
5

Description

## Description
When passing a lazy DataFrame to `LazyPolarsDataset`, it is currently collected into an eager DataFrame before writing it using the appropriate `pl.write_*` function. This can be skipped by writing the lazy dataframe using `pl.sink_*`.

## Context
In some cases, it may be faster to collect the lazy DataFrame in streaming mode. Additionally, it is not always possible to collect the entire DataFrame (e.g., if the data is too large). Using `pl.sink_*`, the entire data set does not need to be loaded.

## Possible Implementation
In the `_save` function, the input DataFrame could first be coerced into a lazy DataFrame and then written to disk using `pl.sink_*`.

## Possible Alternatives

Contributor guide

Open the contributing guide

Research direction

Start at the LazyPolarsDataset._save entry point and inspect how lazy DataFrames are currently collected and passed to pl.write_* functions. Compare the available pl.sink_* APIs for the supported output formats, and confirm that saving avoids loading the entire dataset while preserving existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.