apache / apache/doris

[Bug](zonemap) Float and double zone maps written before has_nan existed are treated as NaN-free

Open
#67,773 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

`ZoneMap::has_nan` is `optional bool = 8` in `ZoneMapPB` (`gensrc/proto/segment_v2.proto`), added by
#55633 together with the NaN handling in the zone map writer. `ZoneMap::from_proto`
(`be/src/storage/index/zone_map/zone_map_index.cpp`) reads it unconditionally, so a zone map
serialized before that field existed deserializes with `has_nan == false`. Nothing checks
`has_has_nan()`, and there is no segment-version gate.

Every consumer that treats `has_nan == false` as "no NaN in this zone" therefore trusts bounds that
may omit one. `range_stats_usable_for_zonemap` (`be/src/exprs/expr_zonemap_filter.cpp`) is one;
#55633 also taught the older olap predicate path to trust the same flag, so this is not limited to
expression zone maps.

Concretely: a FLOAT page written before #55633 holding `{1.0, NaN, 2.0}` stores bounds `[1.0, 2.0]`
with the field absent, because the generic min/max path never selects a NaN. After an upgrade,
`WHERE d > 3.0` prunes the page, but Doris orders NaN above every other value, so that row satisfies
the predicate and should have been returned.

What I have not established is whether any deployment still reads pre-#55633 segments. That decides
the severity, and it is not something the code can answer.

### Solution

Treat a FLOAT or DOUBLE `ZoneMapPB` with field 8 absent as unusable for range pruning, distinguishing
"no NaN" from "unknown", the same way the Parquet readers already distinguish those two with
`floating_nan_count_unknown`. The fix belongs at the `from_proto` or index-meta layer so that every
consumer inherits it.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start in be/src/storage/index/zone_map/zone_map_index.cpp and inspect ZoneMap::from_proto, then trace its use in be/src/exprs/expr_zonemap_filter.cpp and the older OLAP predicate path. Compare the existing Parquet handling of floating_nan_count_unknown; done means FLOAT and DOUBLE zone maps with absent has_nan are excluded from range pruning by every consumer.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.