apache / apache/druid

Plan cannot be built when using virtual column and UNION ALL in SQL query

Open
#10,548 7 comments 3 reactions 0 assignees View on GitHub
Area - SQL Feature
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

# Affected Version

Version: 0.20.0-iap3 (imply build)

### Description

I ran into this problem when playing with sub-query UNION ALL requests, introduced in druid 0.20.0. It was working fine until I had to add a virtual column (`0 as expense`) to have the same columns in the two SELECT.

The easiest way I found to re-create this problem is the following.

```sql
SELECT COUNT(*)
FROM (
(SELECT 1 as a FROM "test-bug")
UNION ALL
(SELECT 2 as a FROM "test-bug")
)
```

```
Error: Unknown exception
Cannot build plan for query: SELECT count(*) FROM ( (SELECT 1 as a FROM "test-bug") UNION ALL (SELECT 2 as a FROM "test-bug") )
org.apache.druid.java.util.common.ISE
```

Stacktrace:
![image](https://user-images.githubusercontent.com/11027160/97791188-e17a4900-1bcf-11eb-8cd8-53e6811b3d21.png)

Note 1: the query works by replacing `SELECT count(*)` by `SELECT a`. And it doesn't work with any expression involving `a` (for eg: `SUM(a)`.
Note 2: The bug is not related to make the union over the same DS. It's just to simply the example. The original bug was found when doing an union between 2 different tables

"test-bug" is a basic data-source created from a kafka-ingestion task.
![image](https://user-images.githubusercontent.com/11027160/97791155-847e9300-1bcf-11eb-8734-2c5f0128d0ce.png)

Note 3: Surprisingly, the same structure works when the union is performed on a system table.

```sql
SELECT COUNT(*)
FROM (
(SELECT 0 as a FROM INFORMATION_SCHEMA.COLUMNS)
UNION ALL
(SELECT -1000 as a FROM INFORMATION_SCHEMA.TABLES)
)
```

Note 4: a real use case that hits this problem is the following:
```sql
SELECT type, SUM(weight) as weight
FROM (
(SELECT type, weight FROM table1)
UNION ALL
(SELECT type, 1 as weight FROM table2) -- weight doesn't exist in table2
)
GROUP BY type
```

Did I touch something not currently supported by druid UNION ALL, or is it a bug in the query transformation?

Thanks

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure using the provided SQL UNION ALL queries with virtual columns and aggregate expressions, comparing them with the working system-table example. Trace the query transformation and planning path associated with these queries; done means the affected COUNT and SUM queries build and execute plans successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.