lance-format / lance-format/lance

Support nested column references in UpdateBuilder::set()

Open
#7,420 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

UpdateBuilder::set() rejects any column name containing ., so a nested field cannot be updated directly — it can only be changed by setting its whole top-level struct column.

Location — rust/lance/src/dataset/write/update.rs, in set():

// TODO: support nested column references. This is mostly blocked on the
// ability to insert them into the RecordBatch properly.
if column.as_ref().contains('.') {
    return Err(Error::not_supported_source(
        format!(
            "Nested column references are not yet supported. Referenced: {}",
            column.as_ref(),
        )
        .into(),
    ));
}
Impact
  • To change s.x, callers must SET s = <whole struct>, which is awkward and rewrites the entire struct value.
  • This is what made #7412 necessary: a whole-struct update resolves to the parent field id, so a scalar index on a nested child field (s.x) had to be handled specially to avoid being wrongly extended over the rewritten fragment.
Ask

Support set("s.x", <expr>) for nested leaf columns. The existing TODO notes this is mostly blocked on inserting nested columns into the RecordBatch properly. Once supported, the index-coverage logic can target the leaf field directly instead of conservatively expanding the whole struct subtree.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance/src/dataset/write/update.rs at UpdateBuilder::set() and trace how update expressions are inserted into a RecordBatch. Study the existing handling of whole-struct updates and the index-coverage logic described in the issue. Done means set("s.x", ) updates the nested leaf directly without rejecting the reference or rewriting the whole struct.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.