apache / apache/kyuubi

[Bug] [flink] Interrupting the JDBC connection cannot stop the engine executing SQL.

Open
#6,519 2 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

The kyuubi connection selection is at the connection level.

When the flink engine is not executing any queries or DML tasks, disconnecting the JDBC connection can automatically shut down the engine.
However, when the flink engine is executing queries or DML tasks, disconnecting the JDBC connection cannot automatically shut down the engine.

Checking the job manager's logs, it can be observed that there is no log for the "close session" request, indicating that the request has not been received.

![image](https://github.com/apache/kyuubi/assets/123344357/62002fdb-51ef-4cca-bd63-d86104c4e53c)

I debugged the kyuubi Flink engine and found that when setting `'table.dml-sync' = 'true'`, the request to interrupt the JDBC connection did not reach the breakpoint while some SQL tasks were still executing. It could reach the breakpoint when there were no SQL tasks running.

It feels like the executing SQL is blocking `TCloseSessionReq`.

![image](https://github.com/apache/kyuubi/assets/123344357/35af5834-b178-46bc-898d-03e1d60a61bd)

The SQL example I executed is as follows:
dml task
```sql
insert overwrite q18v1
SELECT
o_year,
SUM(CASE
WHEN nation = 'BRAZIL' THEN volume
ELSE 0
END) / SUM(volume) AS mkt_share
FROM
(
SELECT
EXTRACT(YEAR FROM o_orderdate) AS o_year,
l_extendedprice * (1 - l_discount) AS volume,
n2.n_name AS nation
FROM
part,
supplier,
lineitem,
orders,
customer,
nation n1,
nation n2,
region
WHERE
p_partkey = l_partkey
AND s_suppkey = l_suppkey
AND l_orderkey = o_orderkey
AND o_custkey = c_custkey
AND c_nationkey = n1.n_nationkey
AND n1.n_regionkey = r_regionkey
AND r_name = 'AMERICA'
AND s_nationkey = n2.n_nationkey
AND o_orderdate BETWEEN DATE '1995-01-01' AND DATE '1996-12-31'
AND p_type = 'ECONOMY ANODIZED STEEL'
) AS all_nations
GROUP BY
o_year
ORDER BY
o_year
```

query task
```sql
SELECT
o_year,
SUM(CASE
WHEN nation = 'BRAZIL' THEN volume
ELSE 0
END) / SUM(volume) AS mkt_share
FROM
(
SELECT
EXTRACT(YEAR FROM o_orderdate) AS o_year,
l_extendedprice * (1 - l_discount) AS volume,
n2.n_name AS nation
FROM
part,
supplier,
lineitem,
orders,
customer,
nation n1,
nation n2,
region
WHERE
p_partkey = l_partkey
AND s_suppkey = l_suppkey
AND l_orderkey = o_orderkey
AND o_custkey = c_custkey
AND c_nationkey = n1.n_nationkey
AND n1.n_regionkey = r_regionkey
AND r_name = 'AMERICA'
AND s_nationkey = n2.n_nationkey
AND o_orderdate BETWEEN DATE '1995-01-01' AND DATE '1996-12-31'
AND p_type = 'ECONOMY ANODIZED STEEL'
) AS all_nations
GROUP BY
o_year
ORDER BY
o_year
```

### Affects Version(s)

1.9.2-snapshot

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

_No response_

### Kyuubi Server Configurations

```yaml
flink.execution.target=yarn-application

kyuubi.engine.session.initialize.sql SET 'table.dml-sync' = 'true';SET 'execution.runtime-mode' = 'batch';
```

### Kyuubi Engine Configurations

_No response_

### Additional context

flink 1.18

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the JDBC disconnect while a Flink SQL task runs with `table.dml-sync` set to `true`, then trace whether the `TCloseSessionReq` reaches the Flink engine. Done means disconnecting the JDBC connection delivers the close-session request and shuts down the engine even while SQL or DML work is executing.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.