matrixorigin / matrixorigin/matrixone
Feature Not Supported: LATERAL derived tables (LATERAL JOINs) not supported
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
MatrixOne does not support LATERAL derived tables (also known as LATERAL JOINs). When using `CROSS JOIN LATERAL` or `LEFT JOIN LATERAL` syntax, MatrixOne returns a SQL syntax error.
## Error Message
```
ERROR 1064 (HY000) at line 11: SQL parser error: You have an error in your SQL syntax; check the manual that corresponds to your MatrixOne server version for the right syntax to use. syntax error at line 4 column 11 near "LATERAL"
```
## Affected Cases
This issue affects **1 test case** from our partition complex testing on the `industry_radar_test` database.
## Related Table DDL
### Table: test_partition_range_lateral
```sql
CREATE TABLE test_partition_range_lateral (
id INT,
company_id INT,
amount DECIMAL(10,2),
created_date DATE
) PARTITION BY RANGE (YEAR(created_date)) (
PARTITION p2022 VALUES LESS THAN (2023),
PARTITION p2023 VALUES LESS THAN (2024),
PARTITION pmax VALUES LESS THAN MAXVALUE
);
```
## Failing SQL Statements
### Example 1: LATERAL JOIN with Partitioned Table
```sql
SELECT c.full_name, tp.amount
FROM company c
CROSS JOIN LATERAL (
SELECT amount
FROM test_partition_range_lateral
WHERE company_id = c.id
AND created_date >= '2022-01-01'
ORDER BY amount DESC
LIMIT 1
) as tp
WHERE c.province = '广东'
LIMIT 10;
```
**Error**: `ERROR 1064 (HY000): SQL parser error: syntax error at line 4 column 11 near "LATERAL"`
### Example 2: LATERAL JOIN with Aggregate
```sql
SELECT c.province, stats.max_value
FROM company c
CROSS JOIN LATERAL (
SELECT MAX(value) as max_value
FROM test_partition_hash_lateral
WHERE company_id = c.id
) as stats
WHERE c.province IS NOT NULL
LIMIT 10;
```
**Error**: `ERROR 1064 (HY000): SQL parser error: syntax error near "LATERAL"`
## Expected Behavior
In MySQL 8.0+, LATERAL derived tables are fully supported. A LATERAL derived table allows the subquery to reference columns from tables that appear earlier in the FROM clause, enabling correlated subqueries in the FROM clause.
Example MySQL syntax:
```sql
SELECT t1.*, t2.*
FROM table1 t1
CROSS JOIN LATERAL (
SELECT *
FROM table2
WHERE table2.id = t1.foreign_id
ORDER BY table2.created_date DESC
LIMIT 1
) as t2;
```
## Impact
- **MySQL Compatibility**: LATERAL JOINs are a standard MySQL 8.0+ feature
- **Migration**: Applications using LATERAL JOINs will fail when migrating to MatrixOne
- **Functionality**: Users cannot write queries that need to reference outer table columns in FROM clause subqueries
- **Query Flexibility**: Limits the ability to write complex correlated queries in the FROM clause
## Use Cases
LATERAL JOINs are commonly used for:
1. **Top-N per Group**: Get the top N records for each group (e.g., latest order per customer)
2. **Correlated Subqueries in FROM**: Reference outer table columns in FROM clause subqueries
3. **Complex Data Transformations**: Perform row-by-row transformations based on related data
4. **Performance Optimization**: Sometimes more efficient than traditional JOINs for certain patterns
## Test Context
This issue was discovered during partition complex testing using the `industry_radar_test` database. The test case attempted to use LATERAL JOIN to get the maximum amount for each company from a partitioned table.
## Workaround
As a temporary workaround, users can rewrite LATERAL JOINs using traditional JOINs or subqueries:
```sql
-- Instead of LATERAL JOIN:
SELECT c.full_name, tp.amount
FROM company c
CROSS JOIN LATERAL (
SELECT amount
FROM test_partition_range_lateral
WHERE company_id = c.id
ORDER BY amount DESC
LIMIT 1
) as tp;
-- Use subquery in SELECT:
SELECT c.full_name,
(SELECT amount
FROM test_partition_range_lateral
WHERE company_id = c.id
ORDER BY amount DESC
LIMIT 1) as amount
FROM company c;
```
However, this workaround:
- May have different performance characteristics
- May not be equivalent in all cases
- Requires query rewriting
## Suggested Fix
1. **Parser Update**: Update the SQL parser to recognize LATERAL keyword in JOIN clauses
2. **Query Planning**: Implement query planner support for LATERAL derived tables
3. **Execution Engine**: Implement execution engine support for LATERAL JOINs
4. **Correlation Handling**: Properly handle column references from outer tables in LATERAL subqueries
5. **Partition Compatibility**: Ensure LATERAL JOINs work correctly with partitioned tables
6. **Documentation**: Update documentation to reflect support for LATERAL JOINs
## Related MySQL Documentation
- [MySQL 8.0 Lateral Derived Tables](https://dev.mysql.com/doc/refman/8.0/en/lateral-derived-tables.html)
Contributor guide
Assessment
This issue has not been assessed yet.