apache / apache/cloudberry

[Bug] ORCA: QD segfault in CExtendedStatsProcessor when extended statistics (dependencies) do not cover all filtered columns

Open
#1,949 0 comments 1 reaction 0 assignees View on GitHub
type: Bug type: Orca
Dominant language
C
Stars
1.4k
Forks
247
Avg merge
4d 3h
Merged PRs (30d)
39

Description

### Apache Cloudberry version

main branch (recent build)

### What happened

With ORCA enabled, the backend crashes when a query filters on a set of columns that is only partially covered by an extended statistics object (`CREATE STATISTICS ... (dependencies)`). In the reproducer below the statistics object covers `c0, c1` and the outer WHERE also references `c2`, which is not covered.

The crash is a null-pointer dereference in the QD backend inside ORCA's statistics derivation:

```
CExtendedStatsProcessor::ApplyCorrelatedStatsToScaleFactorFilterCalculation
(libnaucrates/src/statistics/CExtendedStatsProcessor.cpp, ~line 366)
<- CFilterStatsProcessor::MakeHistHashMapConjFilter
<- CLogicalSelect::PstatsDerive
```

`colid_to_attno_mapping->Find(&colid)` returns `nullptr` for the column that is not part of the statistics object, and the result is dereferenced without a check.

With `optimizer = off` the query runs fine and returns 0 rows.

Found by SQLancer.

### What you think should happen instead

The query should run and return 0 rows (no row has `c1 = ''`). Cardinality estimation must not assume every filtered column is present in the extended statistics object.

### How to reproduce

```sql
CREATE TABLE t3 (c0 boolean, c1 text, c2 int) DISTRIBUTED BY (c0);
INSERT INTO t3 SELECT (g%2=0), 'x'||g, g FROM generate_series(1,100) g;
CREATE STATISTICS s0 (dependencies) ON c0, c1 FROM t3; -- covers c0, c1 only
ANALYZE t3;

SET optimizer = on;
SELECT * FROM (SELECT ALL t3.c0 AS t3c0, t3.c1 AS t3c1, t3.c2 AS t3c2
FROM t3
WHERE (t3.c0) IS TRUE
GROUP BY t3.c0, t3.c1, t3.c2
ORDER BY t3.c1) AS result
WHERE result.t3c0 = TRUE AND result.t3c1 = '' AND result.t3c2 > 0; -- c2 not covered by s0
-- QD backend SIGSEGV
```

### Operating System

Linux

### Anything else

Found while running SQLancer against Cloudberry `main` (3-segment demo cluster, `--enable-cassert`). See the proposal to run SQLancer continuously: https://github.com/apache/cloudberry/discussions/1952

### Are you willing to submit PR?

- [x] Yes, I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Reproduce the SQL example with ORCA enabled and then read libnaucrates/src/statistics/CExtendedStatsProcessor.cpp around CExtendedStatsProcessor::ApplyCorrelatedStatsToScaleFactorFilterCalculation, following the callers through CFilterStatsProcessor::MakeHistHashMapConjFilter and CLogicalSelect::PstatsDerive. Done means the partially covered extended statistics case no longer dereferences a missing column mapping, the backend does not segfault, and the query returns 0 rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.