[FEA] `engine.execute()`: support scanning a persisted result more than once
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
`engine.execute()` (introduced in #23114) returns a `PersistedQueryResult` whose partitions remain GPU-resident on the workers that produced them and can be queried again via `PersistedQueryResult.lazy()`.
Today, persisted results are single-use: scanning a result consumes its partitions, so collecting the same `LazyFrame` twice, or reusing it within a query (for example in a self-join or multi-branch plan), raises an error.
We need persisted results to behave like ordinary query inputs and support multiple scans. This would allow:
- Collecting the same persisted result more than once.
- Reusing a persisted result in self-joins, `concat`, and other multi-branch queries.
- Building multiple downstream queries from the same intermediate result without re-running `engine.execute()`.
Any solution must preserve the streaming runtime's ownership and memory accounting guarantees.
Contributor guide
Assessment
This issue has not been assessed yet.