lance-format / lance-format/lance
Create a logical node for TakeExec
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Create a logical Node for TakeExec.
Design
Parameters of Take:
input: LogicalPlan: The input plan. This must contain a field called<dataset_reference>._rowaddrdataset: Dataset: The dataset to be taking rows from.dataset_reference: TableReference: The name of the relation that is the dataset.dataset_projection: Schema: The Lance schema that should be output from this node. The fields that will be taken are the set difference between the input fields from the dataset<dataset_reference>.*and this projection.
Output schema: ???
Optimization
Take is meant to be run in situations where after scanning, we reduce the cardinality a lot, so it's worth deferring scanning:
TakeExec
FilterExec
ScanExec
or
TakeExec
HashJoinExec
ScanExec
<Other>
That first case is likely better handled by FilteredReadExec. In the second case, I think we need the optimizer to help decide which columns should be handled by take and which by scan.
In necessary_children_exprs, we could either really say that just dataset._rowaddr and any non-dataset columns are needed. That might be okay for a first pass, but eventually we'd like a better deciding factor.
This join case is something we need to solve for #4193.
Implementation
Need to implement UserDefinedLogicalNodeCore. See MergeInsertWriteNode as an example.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the UserDefinedLogicalNodeCore entry point and the MergeInsertWriteNode example. Resolve the intended output schema and necessary child expressions for TakeExec, including how the dataset reference and projection are represented. Done requires an agreed logical-node design and implementation, with the optimizer interaction for the join case considered alongside #4193.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100