Enhanced LIMIT support
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Currently, Beam SQL supports LIMIT in two ways:
1. Within a query, the results are subject to LIMIT. This works.
2. The shell knows to cancel a pipeline when the limit is reached, even if there is unfinished unbounded data.
The canceling of a pipeline works via a basic pattern match against the query execution plan, checking a few child nodes of the BeamEnumerableConverter for a BeamSortRel without a collation. If it can figure out what the limit is for the outermost query, then it will cancel the pipeline.
A more robust approach might be to use traits (or some other thorough analysis) to see if there is a known size for the outermost query. This would, for example, be unaffected by any number of layer of non-size-changing transformations.
Imported from Jira [BEAM-4719](https://issues.apache.org/jira/browse/BEAM-4719). Original Jira may contain additional context.
Reported by: kenn.
Contributor guide
Assessment
This issue has not been assessed yet.