apache / apache/beam

Enhanced LIMIT support

Open
#19,023 0 comments 0 reactions 0 assignees View on GitHub
dsl new feature P3 sql
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Currently, Beam SQL supports LIMIT in two ways:

1. Within a query, the results are subject to LIMIT. This works.
2. The shell knows to cancel a pipeline when the limit is reached, even if there is unfinished unbounded data.

The canceling of a pipeline works via a basic pattern match against the query execution plan, checking a few child nodes of the BeamEnumerableConverter for a BeamSortRel without a collation. If it can figure out what the limit is for the outermost query, then it will cancel the pipeline.

A more robust approach might be to use traits (or some other thorough analysis) to see if there is a known size for the outermost query. This would, for example, be unaffected by any number of layer of non-size-changing transformations.

Imported from Jira [BEAM-4719](https://issues.apache.org/jira/browse/BEAM-4719). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.