lance-format / lance-format/lance
bug: comparing a boolean column with a boolean expression fails to plan when the expression contains a literal
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
With a boolean column on the left of = or !=, the planner converts the literals inside the right-hand expression to the column's type, Boolean, and fails. The same comparison with the operands swapped plans and returns the right rows:
import lance
import pyarrow as pa
ds = lance.write_dataset(
pa.table({"flag": [True, False, True], "id": [1, 0, -1], "x": [0.5, 2.0, 1.0]}),
"bool.lance", mode="overwrite",
)
for f in ["flag != (id > 0)", "flag = (x > 1.0)", "(id > 0) != flag", "flag AND (id > 0)", "flag != TRUE"]:
try:
print(f"{f:22} -> {ds.to_table(filter=f).num_rows} rows")
except Exception as e:
print(f"{f:22} -> error: {' '.join(str(e).split())[:150]}")
On pylance 13.0.0b4 (same on 12.0.0rc1, 11.0.0 and 9.0.0):
flag != (id > 0) -> error: Invalid user input: Error resolving filter expression flag != (id > 0): Invalid user input: Received literal Int64(0) and could not convert to literal
flag = (x > 1.0) -> error: Invalid user input: Error resolving filter expression flag = (x > 1.0): Invalid user input: Received literal Float64(1) and could not convert to liter
(id > 0) != flag -> 1 rows
flag AND (id > 0) -> 1 rows
flag != TRUE -> 1 rows
The full message ends in ...to literal of type 'Boolean'.
Expected
flag != (id > 0) plans like (id > 0) != flag. 0 belongs to id > 0, which already has a type (Int64); it shouldn't be converted to the type of the other side of the outer comparison.
Notes
!=between two booleans is XOR, which filter generators emit for boolean logic; they currently have to spell it as(a AND NOT b) OR (NOT a AND b).- The conversion seems to walk into the whole right-hand operand when only a bare literal operand (
flag != TRUE) should be converted.
Lance version
13.0.0b4
Language binding
Python
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Python reproducer using lance.write_dataset and ds.to_table(filter=...), then trace the filter-expression planner handling comparisons and nested boolean expressions. Done means flag != (id > 0) and flag = (x > 1.0) plan successfully like their operands-swapped equivalents, while bare boolean literals still work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100