lance-format / lance-format/lance

perf: a double-typed literal against a Float32 column casts the column and skips its index

Open
#9,318 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

What happens

An untyped float literal compared with a Float32 column is converted to Float32 and served by the column's scalar index. The same value written as a double (CAST(0.5 AS double), or CAST('-inf' AS double) for a value with no literal syntax) is kept as Float64, so the planner casts the column instead, and the index is no longer used:

import lance
import pyarrow as pa

ds = lance.write_dataset(
    pa.table({"id": list(range(1000)), "x": pa.array([i / 10 for i in range(1000)], pa.float32())}),
    "f32.lance", mode="overwrite",
)
ds.create_scalar_index("x", "BTREE")
ds = lance.dataset("f32.lance")

for f in ["x > 0.5", "x > CAST(0.5 AS double)", "x > CAST(0.5 AS float)", "x < CAST('-inf' AS double)"]:
    plan = ds.scanner(columns=["id"], filter=f).explain_plan()
    full = next(l for l in plan.splitlines() if "full_filter" in l).split("full_filter=")[1].split(", refine_filter")[0]
    print(f"{f:30} index used: {str('ScalarIndexQuery' in plan):5}  planned as: {full}")

On pylance 13.0.0b4 (same on 11.0.0 and 9.0.0):

x > 0.5                        index used: True   planned as: x > Float32(0.5)
x > CAST(0.5 AS double)        index used: False  planned as: CAST(x AS Float64) > Float64(0.5)
x > CAST(0.5 AS float)         index used: True   planned as: x > Float32(0.5)
x < CAST('-inf' AS double)     index used: False  planned as: CAST(x AS Float64) < Float64(-inf)

Results are correct; only the plan differs.

Expected

When a Float64 literal converts to Float32 exactly (as 0.5, -inf and inf do), the comparison is planned against the column, x > Float32(0.5), and uses the index, whichever way the literal was written.

Notes

  • The literal conversion that handles x > 0.5 apparently doesn't apply once the literal carries an explicit type from a folded CAST. Checking whether the value round-trips through f32 before casting the column would cover it.
  • Filter generators hit this naturally: a translator that doesn't track column widths spells special values as CAST('-inf' AS double), which silently disables the index on every Float32 column.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Python reproduction using ds.create_scalar_index("x", "BTREE"), ds.scanner(..., filter=...), and explain_plan() to observe the differing plans for Float32 comparisons. Trace the filter-planning path for explicitly typed double literals and verify that exactly representable values such as 0.5, -inf, and inf keep the comparison on the Float32 column and use ScalarIndexQuery, while results remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.