lance-format / lance-format/lance

feature: accept a float literal against an integer column in filters

Open
#9,317 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

What happens

Comparing an integer column with a float literal fails while resolving the filter, because the literal can't be converted to the column's type without losing precision:

import lance
import pyarrow as pa

ds = lance.write_dataset(pa.table({"id": list(range(1000)), "i": list(range(1000))}), "coerce.lance", mode="overwrite")
ds.create_scalar_index("i", "BTREE")
ds = lance.dataset("coerce.lance")

for f in ["i > 1.5", "CAST(i AS double) > 1.5", "i > 1"]:
    try:
        plan = ds.scanner(columns=["id"], filter=f).explain_plan()
        print(f"{f:25} index used: {'ScalarIndexQuery' in plan}")
    except Exception as e:
        print(f"{f:25} error: {str(e).splitlines()[0]}")

On pylance 13.0.0b4 (same on 11.0.0 and 9.0.0):

i > 1.5                   error: Invalid user input: Error resolving filter expression i > 1.5: Invalid user input: Received literal Float64(1.5) and could not convert to literal of type 'Int64', .../rust/lance-datafusion/src/logical_expr.rs:23:88, .../rust/lance-datafusion/src/planner.rs:950:13
CAST(i AS double) > 1.5   index used: False
i > 1                     index used: True

The only spelling that plans, the explicit cast, loses the scalar index.

Expected

i > 1.5 plans, and uses the index on i. Standard SQL and DataFusion's own type coercion accept it.

Notes

  • safe_coerce_scalar refusing a lossy conversion makes sense as a default (#8847 extended it for Float16). A comparison doesn't need the conversion to be lossless, though: it can be rewritten exactly. i > 1.5i > 1, i >= 1.5i >= 2, i < 1.5i <= 1, i = 1.5 → false (null for null i), i != 1.5i IS NOT NULL, with out-of-range literals clamped. That keeps the comparison on the column and indexable, similar to DataFusion's unwrap-cast-in-comparison rule for integer casts.
  • The same rewrite would let CAST(i AS double) > 1.5 use the index. Clients that generate filters, such as dataframe libraries translating their expressions to Lance SQL, currently have to cast the column and give the index up.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance-datafusion/src/logical_expr.rs and planner.rs, the locations shown in the filter-resolution error, and trace how comparison literals are coerced. Use the Python reproducer from the issue to verify integer-column comparisons plan successfully and retain ScalarIndexQuery usage, including the listed comparison and boundary cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.