lance-format / lance-format/lance
feature: accept a float literal against an integer column in filters
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
What happens
Comparing an integer column with a float literal fails while resolving the filter, because the literal can't be converted to the column's type without losing precision:
import lance
import pyarrow as pa
ds = lance.write_dataset(pa.table({"id": list(range(1000)), "i": list(range(1000))}), "coerce.lance", mode="overwrite")
ds.create_scalar_index("i", "BTREE")
ds = lance.dataset("coerce.lance")
for f in ["i > 1.5", "CAST(i AS double) > 1.5", "i > 1"]:
try:
plan = ds.scanner(columns=["id"], filter=f).explain_plan()
print(f"{f:25} index used: {'ScalarIndexQuery' in plan}")
except Exception as e:
print(f"{f:25} error: {str(e).splitlines()[0]}")
On pylance 13.0.0b4 (same on 11.0.0 and 9.0.0):
i > 1.5 error: Invalid user input: Error resolving filter expression i > 1.5: Invalid user input: Received literal Float64(1.5) and could not convert to literal of type 'Int64', .../rust/lance-datafusion/src/logical_expr.rs:23:88, .../rust/lance-datafusion/src/planner.rs:950:13
CAST(i AS double) > 1.5 index used: False
i > 1 index used: True
The only spelling that plans, the explicit cast, loses the scalar index.
Expected
i > 1.5 plans, and uses the index on i. Standard SQL and DataFusion's own type coercion accept it.
Notes
safe_coerce_scalarrefusing a lossy conversion makes sense as a default (#8847 extended it for Float16). A comparison doesn't need the conversion to be lossless, though: it can be rewritten exactly.i > 1.5→i > 1,i >= 1.5→i >= 2,i < 1.5→i <= 1,i = 1.5→ false (null for nulli),i != 1.5→i IS NOT NULL, with out-of-range literals clamped. That keeps the comparison on the column and indexable, similar to DataFusion's unwrap-cast-in-comparison rule for integer casts.- The same rewrite would let
CAST(i AS double) > 1.5use the index. Clients that generate filters, such as dataframe libraries translating their expressions to Lance SQL, currently have to cast the column and give the index up.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance-datafusion/src/logical_expr.rs and planner.rs, the locations shown in the filter-resolution error, and trace how comparison literals are coerced. Use the Python reproducer from the issue to verify integer-column comparisons plan successfully and retain ScalarIndexQuery usage, including the listed comparison and boundary cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100