lance-format / lance-format/lance

bug: comparing a boolean column with a boolean expression fails to plan when the expression contains a literal

Open
#9,319 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

With a boolean column on the left of = or !=, the planner converts the literals inside the right-hand expression to the column's type, Boolean, and fails. The same comparison with the operands swapped plans and returns the right rows:

import lance
import pyarrow as pa

ds = lance.write_dataset(
    pa.table({"flag": [True, False, True], "id": [1, 0, -1], "x": [0.5, 2.0, 1.0]}),
    "bool.lance", mode="overwrite",
)

for f in ["flag != (id > 0)", "flag = (x > 1.0)", "(id > 0) != flag", "flag AND (id > 0)", "flag != TRUE"]:
    try:
        print(f"{f:22} -> {ds.to_table(filter=f).num_rows} rows")
    except Exception as e:
        print(f"{f:22} -> error: {' '.join(str(e).split())[:150]}")

On pylance 13.0.0b4 (same on 12.0.0rc1, 11.0.0 and 9.0.0):

flag != (id > 0)       -> error: Invalid user input: Error resolving filter expression flag != (id > 0): Invalid user input: Received literal Int64(0) and could not convert to literal
flag = (x > 1.0)       -> error: Invalid user input: Error resolving filter expression flag = (x > 1.0): Invalid user input: Received literal Float64(1) and could not convert to liter
(id > 0) != flag       -> 1 rows
flag AND (id > 0)      -> 1 rows
flag != TRUE           -> 1 rows

The full message ends in ...to literal of type 'Boolean'.

Expected

flag != (id > 0) plans like (id > 0) != flag. 0 belongs to id > 0, which already has a type (Int64); it shouldn't be converted to the type of the other side of the outer comparison.

Notes

  • != between two booleans is XOR, which filter generators emit for boolean logic; they currently have to spell it as (a AND NOT b) OR (NOT a AND b).
  • The conversion seems to walk into the whole right-hand operand when only a bare literal operand (flag != TRUE) should be converted.
Lance version

13.0.0b4

Language binding

Python

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Python reproducer using lance.write_dataset and ds.to_table(filter=...), then trace the filter-expression planner handling comparisons and nested boolean expressions. Done means flag != (id > 0) and flag = (x > 1.0) plan successfully like their operands-swapped equivalents, while bare boolean literals still work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.