lance-format / lance-format/lance

bug: -0.0 and 0.0 still compare unequal between columns and in array_has

Open
#9,316 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

#6236 (fixing #5868) makes filters treat -0.0 and 0.0 as the same value when one side is a literal. Comparisons between two float values, and array_has with a zero, still compare by encoding:

Steps to reproduce
import lance
import pyarrow as pa

data = pa.table({
    "id": [0, 1, 2],
    "a": [-0.0, 0.0, 1.0],
    "b": [0.0, -0.0, 1.0],
    "l": [[-0.0], [0.0], [1.0]],
})
ds = lance.write_dataset(data, "zero.lance", mode="overwrite")

def ids(filter):
    return sorted(ds.to_table(columns=["id"], filter=filter)["id"].to_pylist())

print("a = 0.0            ", ids("a = 0.0"))   # fixed by #6236
print("a = b              ", ids("a = b"))
print("a < b              ", ids("a < b"))
print("array_has(l, 0.0)  ", ids("array_has(l, 0.0)"))
print("array_has(l, -0.0) ", ids("array_has(l, -0.0)"))

On pylance 13.0.0b4:

a = 0.0             [0, 1]
a = b               [2]
a < b               [0]
array_has(l, 0.0)   [1]
array_has(l, -0.0)  [0]
Expected behavior
a = b               [0, 1, 2]
a < b               []
array_has(l, 0.0)   [0, 1]
array_has(l, -0.0)  [0, 1]
Lance version

13.0.0b4

Language binding

Python

Logs / traceback
  • #6236's literal rewrite can't reach a column-to-column comparison. apache/datafusion#22835 normalizes zeros in some DataFusion paths and apache/datafusion#25186 (open) does so for IN lists; the scan filter here still goes through Arrow's total-order kernels.
  • array_has could be rewritten the same way as 0.0 IN (a, b) is in #6236: array_has(l, 0.0)array_has(l, -0.0) OR array_has(l, 0.0).
  • #5868 already points out that -1.0 * 0.0 produces -0.0 in ordinary analytics, which makes column-to-column cases like a * -1 = b realistic.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the provided Python reproduction through ds.to_table(..., filter=...) and focus on the a = b, a < b, and array_has(l, 0.0) entry points. Trace their scan-filter handling into Arrow's total-order kernels; done means both zero encodings compare equally and the results match the expected output for all four cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.