[Bug](zonemap) Float and double zone maps written before has_nan existed are treated as NaN-free
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
`ZoneMap::has_nan` is `optional bool = 8` in `ZoneMapPB` (`gensrc/proto/segment_v2.proto`), added by
#55633 together with the NaN handling in the zone map writer. `ZoneMap::from_proto`
(`be/src/storage/index/zone_map/zone_map_index.cpp`) reads it unconditionally, so a zone map
serialized before that field existed deserializes with `has_nan == false`. Nothing checks
`has_has_nan()`, and there is no segment-version gate.
Every consumer that treats `has_nan == false` as "no NaN in this zone" therefore trusts bounds that
may omit one. `range_stats_usable_for_zonemap` (`be/src/exprs/expr_zonemap_filter.cpp`) is one;
#55633 also taught the older olap predicate path to trust the same flag, so this is not limited to
expression zone maps.
Concretely: a FLOAT page written before #55633 holding `{1.0, NaN, 2.0}` stores bounds `[1.0, 2.0]`
with the field absent, because the generic min/max path never selects a NaN. After an upgrade,
`WHERE d > 3.0` prunes the page, but Doris orders NaN above every other value, so that row satisfies
the predicate and should have been returned.
What I have not established is whether any deployment still reads pre-#55633 segments. That decides
the severity, and it is not something the code can answer.
### Solution
Treat a FLOAT or DOUBLE `ZoneMapPB` with field 8 absent as unusable for range pruning, distinguishing
"no NaN" from "unknown", the same way the Parquet readers already distinguish those two with
`floating_nan_count_unknown`. The fix belongs at the `from_proto` or index-meta layer so that every
consumer inherits it.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Contributor guide
Research direction
Start in be/src/storage/index/zone_map/zone_map_index.cpp and inspect ZoneMap::from_proto, then trace its use in be/src/exprs/expr_zonemap_filter.cpp and the older OLAP predicate path. Compare the existing Parquet handling of floating_nan_count_unknown; done means FLOAT and DOUBLE zone maps with absent has_nan are excluded from range pruning by every consumer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100