pingcap / pingcap/tidb

Planner: parallel_apply + recursive CTE + LATERAL join drops rows from deeper recursion levels

Open
#67,267 0 comments 0 reactions 1 assignee Claimed by @terry1purcell View on GitHub
severity/moderate sig/planner type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

When tidb_enable_parallel_apply=ON, a recursive CTE whose recursive branch uses a LATERAL derived table (cross/inner join) silently drops all rows produced by iterations beyond the first recursive step.

Root cause hypothesis: enableParallelApply in the optimizer sets Concurrency > 1 on the PhysicalApply inside the recursive CTE's recursive branch without checking whether it is in a recursive CTE context. The parallel apply executor likely conflicts with the recursive CTE's working-table iteration model (each CTE iteration must fully drain the previous one before seeding the next), causing later iterations to produce no output.

Workaround: SET tidb_enable_parallel_apply = OFF.

### 1. Minimal reproduce step (Required)

```
CREATE TABLE category (
id INT PRIMARY KEY, parent_id INT, name VARCHAR(50), sort_order INT,
INDEX idx_parent(parent_id, sort_order, name)
);
INSERT INTO category VALUES
(1, NULL, 'root', 0),
(2, 1, 'child_a', 1), (3, 1, 'child_b', 2),
(6, 2, 'grandchild_a1', 1), (7, 2, 'grandchild_a2', 2),
(9, 3, 'grandchild_b1', 1), (10, 3, 'grandchild_b2', 2);
```

### 2. What did you expect to see? (Required)
```
-- Serial (correct):
SET tidb_enable_parallel_apply = OFF;
WITH RECURSIVE tree AS (
SELECT id, name, 1 AS depth FROM category WHERE parent_id IS NULL
UNION ALL
SELECT c.id, c.name, tree.depth + 1
FROM tree CROSS JOIN LATERAL (
SELECT id, name FROM category WHERE parent_id = tree.id ORDER BY sort_order LIMIT 2
) AS c
WHERE tree.depth < 3
)
SELECT id, name, depth FROM tree ORDER BY depth, id;
-- Returns 7 rows: root + 2 children + 4 grandchildren
```
### 3. What did you see instead (Required)
```
-- Parallel (incorrect):
SET tidb_enable_parallel_apply = ON;
-- Same query — returns only 3 rows: root + 2 children; grandchildren are missing
```
Expected: same 7 rows regardless of tidb_enable_parallel_apply.

Actual: with parallel_apply=ON, only 3 rows returned — all rows from the second recursive iteration onward are silently dropped.
### 4. What is your TiDB version? (Required)

Master branch.

Also requires PR https://github.com/pingcap/tidb/pull/67131. However - this enhancement will block parallel apply for recursive apply.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.