apache / apache/kyuubi

[Bug] Kyuubi job status always shows as "Succeeded" even when the actual job has failed

Open
#7,161 3 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

Kyuubi-1.10.2 and spark-3.5.6.
Kyuubi job status always shows as "Succeeded" even when the actual job has failed.
My execution script is as follows:
beeline -u "jdbc:hive2://kyuubiserver.my.com:10009/myuser;principal=hive/kyuubi.my.com@MY.COM;hive.server2.proxy.user=myuser" --hiveconf spark.yarn.queue=center --hiveconf spark.executor.memory=4g -e "insert overwrite directory 'hdfs:///user/myuser/tmp/2025-07-23' select * from ..."

then job faied with error such as:
Caused by: org.apache.kyuubi.plugin.spark.authz.AccessControlException: Permission denied: user [myuser] does not have [write] privilege on ...

But the final job status shown on the Spark log page is "Succeeded", which makes it impossible to determine the actual job outcome based on the job status. Job status monitoring and retry mechanisms become completely unusable.

Image

### Affects Version(s)

1.10.2

### Kyuubi Server Log Output

kyuubi server log:
2025-08-01 10:50:48.340 INFO KyuubiSessionManager-exec-pool: Thread-166 org.apache.kyuubi.operation.ExecuteStatement: Processing myuser's query[71c32a89-bf36-405b-a581-3ca2b2a6202e]: RUNNING_STAT
E -> ERROR_STATE, time taken: 0.439 seconds
2025-08-01 10:50:48.340 INFO KyuubiSessionManager-exec-pool: Thread-166 org.apache.kyuubi.operation.OperationAuditLogger: operation=71c32a89-bf36-405b-a581-3ca2b2a6202e opType=ExecuteSta
tement state=ERROR user=myuser session=5d12d959-c7a6-4dfc-8c18-6a08baf63fbd
2025-08-01 10:50:48.412 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.ExecuteStatement: Processing myuser's query[71c32a89-bf36-405b-a581-3ca2b2a6202e]: ERROR_STA
TE -> CLOSED_STATE, time taken: 0.511 seconds
2025-08-01 10:50:48.412 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.OperationAuditLogger: operation=71c32a89-bf36-405b-a581-3ca2b2a6202e opType=ExecuteSta
tement state=CLOSED user=myuser session=5d12d959-c7a6-4dfc-8c18-6a08baf63fbd
2025-08-01 10:50:48.441 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THand
leIdentifier(guid:71 C3 2A 89 BF 36 40 5B A5 81 3C A2 B2 A6 20 2E, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engin
e side
2025-08-01 10:50:48.471 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [5d12d959-c7a6-4dfc-8
c18-6a08baf63fbd]
2025-08-01 10:50:48.473 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.session.KyuubiSessionManager: myuser's KyuubiSessionImpl with SessionHandle [5d12d959-c7a6-4dfc-8c18-6
a08baf63fbd] is closed, current opening sessions 0
2025-08-01 10:50:48.474 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.LaunchEngine: Processing myuser's query[2af47c76-a3a6-42fe-bd34-4c3aea3933f2]: FINISHED_STAT
E -> CLOSED_STATE, time taken: 34.601 seconds
2025-08-01 10:50:48.474 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.OperationAuditLogger: operation=2af47c76-a3a6-42fe-bd34-4c3aea3933f2 opType=LaunchEngi
ne state=CLOSED user=myuser session=5d12d959-c7a6-4dfc-8c18-6a08baf63fbd
2025-08-01 10:50:48.537 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [5d12d959-c7a6-4dfc-8c18-6a08baf
63fbd]
2025-08-01 10:54:25.523 INFO KyuubiSessionManager-timeout-checker: Thread-63 org.apache.kyuubi.session.KyuubiSessionManager: Checking sessions timeout, current count: 0

### Kyuubi Engine Log Output

```logtalk

```

### Kyuubi Server Configurations

```yaml

```

### Kyuubi Engine Configurations

```yaml

```

### Additional context

_No response_

### Are you willing to submit PR?

- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the status flow through ExecuteStatement and LaunchEngine, using the reported ERROR_STATE and FINISHED_STATE log transitions alongside the Spark log page result. Reproduce the failed authorization case and verify that the displayed job status reflects the failed operation rather than Succeeded, with regression coverage for the failure path.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark, sql
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.