lance-format / lance-format/lance
bug: -0.0 and 0.0 still compare unequal between columns and in array_has
Open
Nobody has claimed this yet.
bug
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
#6236 (fixing #5868) makes filters treat -0.0 and 0.0 as the same value when one side is a literal. Comparisons between two float values, and array_has with a zero, still compare by encoding:
Steps to reproduce
import lance
import pyarrow as pa
data = pa.table({
"id": [0, 1, 2],
"a": [-0.0, 0.0, 1.0],
"b": [0.0, -0.0, 1.0],
"l": [[-0.0], [0.0], [1.0]],
})
ds = lance.write_dataset(data, "zero.lance", mode="overwrite")
def ids(filter):
return sorted(ds.to_table(columns=["id"], filter=filter)["id"].to_pylist())
print("a = 0.0 ", ids("a = 0.0")) # fixed by #6236
print("a = b ", ids("a = b"))
print("a < b ", ids("a < b"))
print("array_has(l, 0.0) ", ids("array_has(l, 0.0)"))
print("array_has(l, -0.0) ", ids("array_has(l, -0.0)"))
On pylance 13.0.0b4:
a = 0.0 [0, 1]
a = b [2]
a < b [0]
array_has(l, 0.0) [1]
array_has(l, -0.0) [0]
Expected behavior
a = b [0, 1, 2]
a < b []
array_has(l, 0.0) [0, 1]
array_has(l, -0.0) [0, 1]
Lance version
13.0.0b4
Language binding
Python
Logs / traceback
- #6236's literal rewrite can't reach a column-to-column comparison. apache/datafusion#22835 normalizes zeros in some DataFusion paths and apache/datafusion#25186 (open) does so for
INlists; the scan filter here still goes through Arrow's total-order kernels. array_hascould be rewritten the same way as0.0 IN (a, b)is in #6236:array_has(l, 0.0)→array_has(l, -0.0) OR array_has(l, 0.0).- #5868 already points out that
-1.0 * 0.0produces-0.0in ordinary analytics, which makes column-to-column cases likea * -1 = brealistic.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the provided Python reproduction through ds.to_table(..., filter=...) and focus on the a = b, a < b, and array_has(l, 0.0) entry points. Trace their scan-filter handling into Arrow's total-order kernels; done means both zero encodings compare equally and the results match the expected output for all four cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100