apache / apache/doris

[Bug] NullPointerException in RewriteCteChildren when processing complex nested CTEs with set operations

Open
#57,348 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

2.1.6 & master(latest)

### What's Wrong?
A NullPointerException occurs when executing queries with nested CTEs that contain set operations (EXCEPT/UNION) combined with an unused CTE that references other CTEs via scalar subqueries.
```
Caused by: org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Cannot invoke "java.util.Set.size()" because "producerOutputs" is null
... 12 more
Caused by: java.lang.NullPointerException: Cannot invoke "java.util.Set.size()" because "producerOutputs" is null
at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEProducer(RewriteCteChildren.java:118) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEProducer(RewriteCteChildren.java:57) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer.accept(LogicalCTEProducer.java:74) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.visitLogicalCTEAnchor(RewriteCteChildren.java:57) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor.accept(LogicalCTEAnchor.java:62) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.rules.rewrite.RewriteCteChildren.rewriteRoot(RewriteCteChildren.java:68) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.jobs.rewrite.CustomRewriteJob.execute(CustomRewriteJob.java:61) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:139) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.NereidsPlanner.lambda$rewrite$4(NereidsPlanner.java:321) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.NereidsPlanner.keepOrShowPlanProcess(NereidsPlanner.java:703) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.NereidsPlanner.rewrite(NereidsPlanner.java:321) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.NereidsPlanner.planWithoutLock(NereidsPlanner.java:237) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:202) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:127) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.commands.ExplainCommand.run(ExplainCommand.java:88) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:718) ~[doris-fe.jar:1.2-SNAPSHOT]
... 11 more
```

### What You Expected?

The query should execute successfully, with unused CTEs eliminated gracefully during optimization without causing NPE.

### How to Reproduce?

### Table DDL
```sql
CREATE TABLE `test_sales`(
`id` int,
`item` string,
`amount` double)
PARTITIONED BY (
`year` int,
`month` int)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
WITH SERDEPROPERTIES (
'serialization.format' = '1')
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
```

### Minimal case to reproduce
```sql
with
tbl1 as (select amount from test_sales where year = 2023 EXCEPT select amount from test_sales where year = 2024),
tbl2 as (select amount from test_sales where year = 2024 EXCEPT select amount from test_sales where year = 2025),
total as (select (select count(1) from tbl1), (select count(1) from tbl2))
select * from tbl1 UNION ALL select * from tbl2;
```

### Anything Else?

Workaround: The NPE can be avoided by disabling the following rule:
SET global disable_nereids_rules='CLEAR_CONTEXT_STATUS'

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.