NVIDIA / NVIDIA/cudf

[FEA] `engine.execute()`: support scanning a persisted result more than once

Open
#23,115 0 comments 0 reactions 1 assignee Claimed by @madsbk View on GitHub
feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

`engine.execute()` (introduced in #23114) returns a `PersistedQueryResult` whose partitions remain GPU-resident on the workers that produced them and can be queried again via `PersistedQueryResult.lazy()`.

Today, persisted results are single-use: scanning a result consumes its partitions, so collecting the same `LazyFrame` twice, or reusing it within a query (for example in a self-join or multi-branch plan), raises an error.

We need persisted results to behave like ordinary query inputs and support multiple scans. This would allow:

- Collecting the same persisted result more than once.
- Reusing a persisted result in self-joins, `concat`, and other multi-branch queries.
- Building multiple downstream queries from the same intermediate result without re-running `engine.execute()`.

Any solution must preserve the streaming runtime's ownership and memory accounting guarantees.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.