matrixorigin / matrixorigin/matrixone

MySQL Compatibility: JOIN column reference remapping error in nested correlated subqueries

Open
#23,083 1 comment 0 reactions 1 assignee Assigned to @XuPeng-SH View on GitHub
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

MatrixOne encounters a query optimizer error when processing nested correlated subqueries in SELECT clauses. The error occurs when a correlated subquery contains an IN subquery, causing the query optimizer to fail in mapping column references during JOIN operations.

## Error Message

```
ERROR 1064 (HY000) at line 1: SQL parser error: remapInfo step 0 nodeId 11 nodeType JOIN tip OnList colRefCnt:[[12 0] : 0][[18 1] : 0][[18 3] : 0][[3 0] : 1][[3 1] : 1][[3 2] : 1][[1 0] : 1][[2 0] : 1][[6 21] : 0][[7 0] : 0][[20 0] : 0][[8 0] : 2][[20 1] : 0][[12 21] : 0]colRefBool:[[0 1] : true][[0 2] : true][[0 0] : true]sinkColRef:ColRefRemapping{globalToLocallocalToGlobal}empty ColRefRemapping srcExprIdx 0 ; can't find column [6 21] in context's map { [1 0], [2 0], [7 0], [8 0] }
```

## Affected Cases

This issue affects **2 test cases** from our nested subquery compatibility testing on the `industry_radar_test` database.

## Related Table DDL

### Table: company

```sql
CREATE TABLE `company` (
`id` int NOT NULL,
`full_name` varchar(200) DEFAULT NULL,
`short_name` varchar(128) DEFAULT NULL,
`update_time` datetime NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp() COMMENT '最后修改时间',
`address` varchar(255) DEFAULT NULL,
`board_id` int DEFAULT NULL,
`business_scope` text DEFAULT NULL,
`city` varchar(32) DEFAULT NULL,
`company_profile` text DEFAULT NULL,
`company_type_tags` varchar(255) DEFAULT NULL,
`country` varchar(20) DEFAULT NULL,
`directory_label` varchar(512) DEFAULT NULL,
`district` varchar(32) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`english_name` varchar(255) DEFAULT NULL,
`enterprises_type` varchar(50) DEFAULT NULL,
`founded_time` datetime(6) DEFAULT NULL,
`industry` varchar(32) DEFAULT NULL,
`industry_paqu` varchar(1000) DEFAULT NULL,
`insured_num` varchar(32) DEFAULT NULL,
`product_label` varchar(512) DEFAULT NULL,
`province` varchar(32) DEFAULT NULL,
`registered_capital` varchar(50) DEFAULT NULL,
`representative` varchar(32) DEFAULT NULL,
`taxpayer_num` varchar(50) DEFAULT NULL,
`telephone` varchar(255) DEFAULT NULL,
`website` varchar(500) DEFAULT NULL,
`industry_web_label` varchar(512) DEFAULT NULL COMMENT '国标行业标签',
`industry_domain_label` varchar(512) DEFAULT NULL COMMENT '素问产业链标签',
`scale` varchar(255) DEFAULT NULL COMMENT '规模标签',
`policy_type` varchar(512) DEFAULT NULL COMMENT '政策类型标签',
PRIMARY KEY (`id`),
UNIQUE KEY `company_full_name_idx` (`full_name`),
UNIQUE KEY `company_short_name_idx` (`short_name`),
KEY `company_board_id_idx` (`board_id`),
KEY `company_city_idx` (`city`),
KEY `company_province_idx` (`province`)
)
```

### Table: company_patent

```sql
CREATE TABLE `company_patent` (
`id` int NOT NULL,
`company_id` int DEFAULT NULL,
`name` varchar(512) DEFAULT NULL,
`status` varchar(50) DEFAULT NULL,
`patent_type` varchar(50) DEFAULT NULL,
`num` varchar(100) DEFAULT NULL,
`public_date` date DEFAULT NULL,
`application_date` date DEFAULT NULL,
`update_time` datetime NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp() COMMENT '最后修改时间',
PRIMARY KEY (`id`),
KEY `company_id_idx` (`company_id`),
KEY `status_idx` (`status`)
)
```

### Table: news_company

```sql
CREATE TABLE `news_company` (
`id` int NOT NULL,
`company_id` int DEFAULT NULL,
`news_id` int DEFAULT NULL,
`release_time` datetime DEFAULT NULL,
`extract_type` varchar(50) DEFAULT NULL,
`is_hot` smallint DEFAULT NULL,
`polarity` smallint DEFAULT NULL,
`update_time` datetime NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp() COMMENT '最后修改时间',
PRIMARY KEY (`id`),
KEY `company_id_idx` (`company_id`),
KEY `news_id_idx` (`news_id`)
)
```

## Failing SQL Statements

### Example 1: Correlated subquery with IN subquery in SELECT

```sql
SELECT c.province,
(SELECT COUNT(DISTINCT c2.id)
FROM company c2
WHERE c2.province = c.province
AND c2.id IN (
SELECT company_id FROM company_patent WHERE status = '授权'
)
) as companies_with_patents,
COUNT(*) as total_companies
FROM company c
WHERE c.province IS NOT NULL
GROUP BY c.province
LIMIT 10;
```

**Error**: `ERROR 1064 (HY000): SQL parser error: remapInfo step 0 nodeId 11 nodeType JOIN tip OnList ... can't find column [6 21] in context's map`

### Example 2: JOIN with derived table containing correlated subquery

```sql
SELECT c.full_name,
subq.patent_count,
subq.news_count
FROM company c
INNER JOIN (
SELECT cp.company_id,
COUNT(cp.id) as patent_count,
(SELECT COUNT(*)
FROM news_company nc
WHERE nc.company_id = cp.company_id
) as news_count
FROM company_patent cp
WHERE cp.company_id IN (
SELECT id FROM company WHERE province = '广东'
)
GROUP BY cp.company_id
) as subq ON c.id = subq.company_id
LIMIT 10;
```

**Error**: `ERROR 1064 (HY000): SQL parser error: remapInfo step 0 nodeId 12 nodeType JOIN tip OnList ... can't find column [12 1] in context's map`

## Expected Behavior

In MySQL, nested correlated subqueries with IN subqueries in SELECT clauses execute successfully. The query optimizer should correctly map column references across all levels of nesting.

## Impact

- **MySQL Compatibility**: This is a standard MySQL feature that should work correctly
- **Migration**: Applications using nested correlated subqueries will fail when migrating to MatrixOne
- **Functionality**: Users cannot use correlated subqueries with nested IN subqueries in SELECT clauses
- **Query Complexity**: Limits the complexity of analytical queries that can be executed

## Root Cause Analysis

The error occurs in the query optimizer's column reference remapping logic:
1. When processing JOIN operations in nested subqueries
2. The optimizer fails to correctly map column references (`[6 21]`, `[12 1]`) to the context map
3. This suggests an issue in the `remapInfo` logic during query plan generation
4. The problem appears to be in how column references are tracked across nested query levels

## Test Context

This issue was discovered during nested subquery compatibility testing using the `industry_radar_test` database, which contains:
- `company` table with ~766K records
- `company_patent` table with ~1.2M records
- `news_company` table with ~1.5M records

## Suggested Fix

1. **Query Optimizer**: Review and fix the column reference remapping logic in the query optimizer
2. **Column Reference Tracking**: Ensure column references are correctly tracked across nested query levels
3. **JOIN Processing**: Fix the JOIN column reference mapping when processing nested correlated subqueries
4. **Error Handling**: Improve error messages to provide more context about which column reference failed
5. **Testing**: Add comprehensive tests for nested correlated subqueries with IN subqueries
6. **Code Review**: Review `remapInfo` logic in the query optimizer, particularly for JOIN operations with nested subqueries

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.