matrixorigin / matrixorigin/matrixone

Feature Not Supported: LATERAL derived tables (LATERAL JOINs) not supported

Open
#23,099 1 comment 0 reactions 1 assignee Claimed by @XuPeng-SH View on GitHub
kind/feature priority/p1
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

MatrixOne does not support LATERAL derived tables (also known as LATERAL JOINs). When using `CROSS JOIN LATERAL` or `LEFT JOIN LATERAL` syntax, MatrixOne returns a SQL syntax error.

## Error Message

```
ERROR 1064 (HY000) at line 11: SQL parser error: You have an error in your SQL syntax; check the manual that corresponds to your MatrixOne server version for the right syntax to use. syntax error at line 4 column 11 near "LATERAL"
```

## Affected Cases

This issue affects **1 test case** from our partition complex testing on the `industry_radar_test` database.

## Related Table DDL

### Table: test_partition_range_lateral

```sql
CREATE TABLE test_partition_range_lateral (
id INT,
company_id INT,
amount DECIMAL(10,2),
created_date DATE
) PARTITION BY RANGE (YEAR(created_date)) (
PARTITION p2022 VALUES LESS THAN (2023),
PARTITION p2023 VALUES LESS THAN (2024),
PARTITION pmax VALUES LESS THAN MAXVALUE
);
```

## Failing SQL Statements

### Example 1: LATERAL JOIN with Partitioned Table

```sql
SELECT c.full_name, tp.amount
FROM company c
CROSS JOIN LATERAL (
SELECT amount
FROM test_partition_range_lateral
WHERE company_id = c.id
AND created_date >= '2022-01-01'
ORDER BY amount DESC
LIMIT 1
) as tp
WHERE c.province = '广东'
LIMIT 10;
```

**Error**: `ERROR 1064 (HY000): SQL parser error: syntax error at line 4 column 11 near "LATERAL"`

### Example 2: LATERAL JOIN with Aggregate

```sql
SELECT c.province, stats.max_value
FROM company c
CROSS JOIN LATERAL (
SELECT MAX(value) as max_value
FROM test_partition_hash_lateral
WHERE company_id = c.id
) as stats
WHERE c.province IS NOT NULL
LIMIT 10;
```

**Error**: `ERROR 1064 (HY000): SQL parser error: syntax error near "LATERAL"`

## Expected Behavior

In MySQL 8.0+, LATERAL derived tables are fully supported. A LATERAL derived table allows the subquery to reference columns from tables that appear earlier in the FROM clause, enabling correlated subqueries in the FROM clause.

Example MySQL syntax:
```sql
SELECT t1.*, t2.*
FROM table1 t1
CROSS JOIN LATERAL (
SELECT *
FROM table2
WHERE table2.id = t1.foreign_id
ORDER BY table2.created_date DESC
LIMIT 1
) as t2;
```

## Impact

- **MySQL Compatibility**: LATERAL JOINs are a standard MySQL 8.0+ feature
- **Migration**: Applications using LATERAL JOINs will fail when migrating to MatrixOne
- **Functionality**: Users cannot write queries that need to reference outer table columns in FROM clause subqueries
- **Query Flexibility**: Limits the ability to write complex correlated queries in the FROM clause

## Use Cases

LATERAL JOINs are commonly used for:
1. **Top-N per Group**: Get the top N records for each group (e.g., latest order per customer)
2. **Correlated Subqueries in FROM**: Reference outer table columns in FROM clause subqueries
3. **Complex Data Transformations**: Perform row-by-row transformations based on related data
4. **Performance Optimization**: Sometimes more efficient than traditional JOINs for certain patterns

## Test Context

This issue was discovered during partition complex testing using the `industry_radar_test` database. The test case attempted to use LATERAL JOIN to get the maximum amount for each company from a partitioned table.

## Workaround

As a temporary workaround, users can rewrite LATERAL JOINs using traditional JOINs or subqueries:

```sql
-- Instead of LATERAL JOIN:
SELECT c.full_name, tp.amount
FROM company c
CROSS JOIN LATERAL (
SELECT amount
FROM test_partition_range_lateral
WHERE company_id = c.id
ORDER BY amount DESC
LIMIT 1
) as tp;

-- Use subquery in SELECT:
SELECT c.full_name,
(SELECT amount
FROM test_partition_range_lateral
WHERE company_id = c.id
ORDER BY amount DESC
LIMIT 1) as amount
FROM company c;
```

However, this workaround:
- May have different performance characteristics
- May not be equivalent in all cases
- Requires query rewriting

## Suggested Fix

1. **Parser Update**: Update the SQL parser to recognize LATERAL keyword in JOIN clauses
2. **Query Planning**: Implement query planner support for LATERAL derived tables
3. **Execution Engine**: Implement execution engine support for LATERAL JOINs
4. **Correlation Handling**: Properly handle column references from outer tables in LATERAL subqueries
5. **Partition Compatibility**: Ensure LATERAL JOINs work correctly with partitioned tables
6. **Documentation**: Update documentation to reflect support for LATERAL JOINs

## Related MySQL Documentation

- [MySQL 8.0 Lateral Derived Tables](https://dev.mysql.com/doc/refman/8.0/en/lateral-derived-tables.html)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.