matrixorigin / matrixorigin/matrixone
Feature Not Supported: Recursive CTE cannot query partitioned tables in recursive member
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
MatrixOne does not support using partitioned tables in the recursive part of a Recursive Common Table Expression (CTE). When a recursive CTE attempts to query a partitioned table in the recursive member (the part after `UNION ALL`), MatrixOne returns a SQL syntax error.
## Error Message
```
ERROR 1064 (HY000) at line 11: SQL parser error: You have an error in your SQL syntax; check the manual that corresponds to your MatrixOne server version for the right syntax to use. syntax error at line 7 column 10 near "UNION ALL"
```
## Affected Cases
This issue affects **1 test case** from our partition complex testing on the `industry_radar_test` database.
## Related Table DDL
### Table: test_partition_range_recursive
```sql
CREATE TABLE test_partition_range_recursive (
id INT,
company_id INT,
parent_id INT,
created_date DATE
) PARTITION BY RANGE (YEAR(created_date)) (
PARTITION p2022 VALUES LESS THAN (2023),
PARTITION p2023 VALUES LESS THAN (2024),
PARTITION pmax VALUES LESS THAN MAXVALUE
);
```
## Failing SQL Statements
### Example 1: Recursive CTE with Partitioned Table in Recursive Part
```sql
WITH RECURSIVE company_hierarchy AS (
SELECT company_id, parent_id, 1 as level
FROM test_partition_range_recursive
WHERE parent_id IS NULL
AND created_date >= '2022-01-01'
LIMIT 10
UNION ALL
SELECT tp.company_id, tp.parent_id, ch.level + 1
FROM test_partition_range_recursive tp
INNER JOIN company_hierarchy ch ON tp.parent_id = ch.company_id
WHERE tp.created_date >= '2022-01-01'
AND ch.level < 5
)
SELECT * FROM company_hierarchy LIMIT 10;
```
**Error**: `ERROR 1064 (HY000): SQL parser error: syntax error at line 7 column 10 near "UNION ALL"`
**Note**: The error occurs specifically when the recursive member (after `UNION ALL`) references a partitioned table.
## Expected Behavior
In MySQL 8.0+, Recursive CTEs are fully supported and can reference partitioned tables in both the anchor member (initial SELECT) and the recursive member (after UNION ALL). The recursive CTE should be able to traverse hierarchical data stored in partitioned tables.
Example MySQL syntax:
```sql
WITH RECURSIVE hierarchy AS (
-- Anchor member: can use partitioned table
SELECT id, parent_id, 1 as level
FROM partitioned_table
WHERE parent_id IS NULL
UNION ALL
-- Recursive member: should also be able to use partitioned table
SELECT pt.id, pt.parent_id, h.level + 1
FROM partitioned_table pt
INNER JOIN hierarchy h ON pt.parent_id = h.id
WHERE h.level < 10
)
SELECT * FROM hierarchy;
```
## Impact
- **MySQL Compatibility**: Recursive CTEs with partitioned tables are a standard MySQL 8.0+ feature
- **Migration**: Applications using recursive CTEs on partitioned tables will fail when migrating to MatrixOne
- **Functionality**: Users cannot traverse hierarchical data stored in partitioned tables using recursive CTEs
- **Query Flexibility**: Limits the ability to perform recursive queries on partitioned data
## Use Cases
Recursive CTEs with partitioned tables are commonly used for:
1. **Hierarchical Data**: Traverse organizational hierarchies, category trees, or comment threads stored in partitioned tables
2. **Graph Traversal**: Find paths or relationships in graph-like data structures
3. **Time-Series Analysis**: Recursively process time-series data stored in date-partitioned tables
4. **Data Aggregation**: Recursively aggregate data across partition boundaries
## Test Context
This issue was discovered during partition complex testing using the `industry_radar_test` database. The test case attempted to use a recursive CTE to traverse a company hierarchy stored in a date-partitioned table.
## Important Notes
1. **Non-Recursive CTEs Work**: Non-recursive CTEs (regular CTEs) work correctly with partitioned tables. The issue is specific to the recursive part of recursive CTEs.
2. **Anchor Member Works**: The anchor member (initial SELECT) of a recursive CTE can successfully query partitioned tables. The error occurs only in the recursive member (after UNION ALL).
3. **Previous Issues**: This is related to but distinct from previous recursive CTE issues:
- Issue #8: GROUP BY not supported in recursive CTE
- Issue #9: LEFT/RIGHT/OUTER JOIN not supported in recursive CTE
- Issue #10: ORDER BY/LIMIT not supported in recursive CTE anchor part
## Workaround
As a temporary workaround, users can:
1. Use non-partitioned tables for recursive CTEs
2. Use application-level recursion instead of SQL recursive CTEs
3. Flatten the hierarchy into a non-partitioned table before using recursive CTEs
However, these workarounds:
- May have performance implications
- May require data migration
- May not be equivalent in all cases
## Suggested Fix
1. **Parser Update**: Update the SQL parser to allow partitioned table references in the recursive member of recursive CTEs
2. **Query Planning**: Ensure the query planner can handle partitioned tables in recursive CTE execution plans
3. **Execution Engine**: Implement execution engine support for recursive CTEs that query partitioned tables
4. **Partition Pruning**: Ensure partition pruning works correctly in recursive CTE context
5. **Testing**: Add comprehensive tests for recursive CTEs with various partition types (RANGE, HASH, KEY)
6. **Documentation**: Update documentation to clarify support for recursive CTEs with partitioned tables
## Related MySQL Documentation
- [MySQL 8.0 Recursive Common Table Expressions](https://dev.mysql.com/doc/refman/8.0/en/with.html#common-table-expressions-recursive)
- [MySQL Partitioning](https://dev.mysql.com/doc/refman/8.0/en/partitioning.html)
Contributor guide
Assessment
This issue has not been assessed yet.