lance-format / lance-format/lance
perf: a double-typed literal against a Float32 column casts the column and skips its index
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
What happens
An untyped float literal compared with a Float32 column is converted to Float32 and served by the column's scalar index. The same value written as a double (CAST(0.5 AS double), or CAST('-inf' AS double) for a value with no literal syntax) is kept as Float64, so the planner casts the column instead, and the index is no longer used:
import lance
import pyarrow as pa
ds = lance.write_dataset(
pa.table({"id": list(range(1000)), "x": pa.array([i / 10 for i in range(1000)], pa.float32())}),
"f32.lance", mode="overwrite",
)
ds.create_scalar_index("x", "BTREE")
ds = lance.dataset("f32.lance")
for f in ["x > 0.5", "x > CAST(0.5 AS double)", "x > CAST(0.5 AS float)", "x < CAST('-inf' AS double)"]:
plan = ds.scanner(columns=["id"], filter=f).explain_plan()
full = next(l for l in plan.splitlines() if "full_filter" in l).split("full_filter=")[1].split(", refine_filter")[0]
print(f"{f:30} index used: {str('ScalarIndexQuery' in plan):5} planned as: {full}")
On pylance 13.0.0b4 (same on 11.0.0 and 9.0.0):
x > 0.5 index used: True planned as: x > Float32(0.5)
x > CAST(0.5 AS double) index used: False planned as: CAST(x AS Float64) > Float64(0.5)
x > CAST(0.5 AS float) index used: True planned as: x > Float32(0.5)
x < CAST('-inf' AS double) index used: False planned as: CAST(x AS Float64) < Float64(-inf)
Results are correct; only the plan differs.
Expected
When a Float64 literal converts to Float32 exactly (as 0.5, -inf and inf do), the comparison is planned against the column, x > Float32(0.5), and uses the index, whichever way the literal was written.
Notes
- The literal conversion that handles
x > 0.5apparently doesn't apply once the literal carries an explicit type from a foldedCAST. Checking whether the value round-trips throughf32before casting the column would cover it. - Filter generators hit this naturally: a translator that doesn't track column widths spells special values as
CAST('-inf' AS double), which silently disables the index on everyFloat32column.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Python reproduction using ds.create_scalar_index("x", "BTREE"), ds.scanner(..., filter=...), and explain_plan() to observe the differing plans for Float32 comparisons. Trace the filter-planning path for explicitly typed double literals and verify that exactly representable values such as 0.5, -inf, and inf keep the comparison on the Float32 column and use ScalarIndexQuery, while results remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100