lance-format / lance-format/lance

Create a logical node for TakeExec

Open
#4,266 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Create a logical Node for TakeExec.

Design

Parameters of Take:

  • input: LogicalPlan: The input plan. This must contain a field called <dataset_reference>._rowaddr
  • dataset: Dataset: The dataset to be taking rows from.
  • dataset_reference: TableReference: The name of the relation that is the dataset.
  • dataset_projection: Schema: The Lance schema that should be output from this node. The fields that will be taken are the set difference between the input fields from the dataset <dataset_reference>.* and this projection.

Output schema: ???

Optimization

Take is meant to be run in situations where after scanning, we reduce the cardinality a lot, so it's worth deferring scanning:

TakeExec
  FilterExec
    ScanExec

or

TakeExec
  HashJoinExec
    ScanExec
    <Other>

That first case is likely better handled by FilteredReadExec. In the second case, I think we need the optimizer to help decide which columns should be handled by take and which by scan.

In necessary_children_exprs, we could either really say that just dataset._rowaddr and any non-dataset columns are needed. That might be okay for a first pass, but eventually we'd like a better deciding factor.

This join case is something we need to solve for #4193.

Implementation

Need to implement UserDefinedLogicalNodeCore. See MergeInsertWriteNode as an example.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the UserDefinedLogicalNodeCore entry point and the MergeInsertWriteNode example. Resolve the intended output schema and necessary child expressions for TakeExec, including how the dataset reference and projection are represented. Done requires an agreed logical-node design and implementation, with the optimizer interaction for the join case considered alongside #4193.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.