bug: a NaN with its sign bit set sorts below every number in filters

Description

Float filters order values by IEEE 754 total order, where a NaN with its sign bit set comes before -inf. SQL engines (PostgreSQL treats every NaN as equal and greater than any number), IEEE 754 comparisons, and dataframe libraries such as Polars don't order NaN that way. A negative NaN is not unusual either: on x86 it is the NaN produced by 0.0 / 0.0, inf - inf, sqrt(-1) and inf * 0 in numpy, pyarrow and Polars, and by negating a NaN.

Steps to reproduce

So the same logical value answers differently depending on its sign bit:

import math
import lance
import pyarrow as pa

neg_nan = math.copysign(math.nan, -1.0)  # what 0.0/0.0 and inf-inf produce on x86
data = pa.table({"id": [0, 1, 2, 3], "x": [neg_nan, math.nan, 1.0, 2.0]})
ds = lance.write_dataset(data, "nan.lance", mode="overwrite")

def ids(filter):
    return sorted(ds.to_table(columns=["id"], filter=filter)["id"].to_pylist())

print("x > 1.0      ", ids("x > 1.0"))
print("x < 1.0      ", ids("x < 1.0"))
print("isnan(x)     ", ids("isnan(x)"))
ds.create_scalar_index("x", "BTREE")
ds = lance.dataset("nan.lance")
print("x > 1.0 btree", ids("x > 1.0"))

On pylance 13.0.0b4 (same on 11.0.0 and 9.0.0):

x > 1.0       [1, 3]
x < 1.0       [0]
isnan(x)      [0, 1]
x > 1.0 btree [1, 3]

Rows 0 and 1 are both NaN (isnan agrees), yet x > 1.0 keeps only the positive one and x < 1.0 keeps the negative one. A column-to-column comparison is affected the same way: a negative and a positive NaN compare unequal, and the negative one compares less than any number. The BTREE index agrees with the scan, so the result is consistent across plans, just not what the data means.

Expected behavior

The sign of a NaN doesn't affect comparisons: x > 1.0 returns [1, 3] plus row 0, and x < 1.0 returns neither NaN.

Lance version

13.0.0b4

Language binding

Python

Notes

• Same root cause as #5868: Arrow's comparison kernels use total_cmp and ask callers to normalize. #6236 fixed this for signed zeros by rewriting literals in the planner; NaN signs aren't covered.
• DataFusion's IN list work (apache/datafusion#25186) deliberately keeps distinct NaN payloads distinct, so this probably has to be handled in Lance, as #6236 did for zeros.
• A literal-side rewrite is possible without losing the index, since x < CAST('-inf' AS double) holds for exactly the negative NaNs: x > c → x > c OR x < -inf, x < c → x < c AND x >= -inf. Normalizing NaN to one sign on write, or in the comparison, would also cover column-to-column comparisons.

Start with the planner's signed-zero literal rewrite from #6236, then inspect how Arrow comparison kernels use total_cmp and how BTREE filtering applies the predicates. Reproduce the negative-NaN cases from the issue and verify that scans and indexed queries treat NaNs consistently with the expected comparisons, including column-to-column comparisons.