apache / apache/kyuubi

[Bug] The Kyuubi query did not take effect when entering the ERROR_STATE from RUNNING-STATE

Open
#7,314 1 comment 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

The Kyuubi query did not take effect when entering the ERROR_STATE from RUNNING-STATE, and eventually changed back to RUNNING-STATE. Then keep running continuously until timeout or manual kill. The log show below.
```
: 26/01/25 04:24:11 INFO[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Processing g10_us's query[284c07fb-a061-4489-a725-e3d68b2174d9]: RUNNING_STATE -> ERROR_STATE, time taken: 7.146 seconds"

: 26/01/25 04:24:11 ERROR[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Exception in query[284c07fb-a061-4489-a725-e3d68b2174d9]: Error operating ExecuteStatement:[ org.apache.hadoop.mapred.InvalidInputException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.hadoop.mapred.InvalidInputException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124

: Caused by:[ java.io.IOException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Fjava.io.IOException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124

2026-01-25 04:24:13.942 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:18.943 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:23.944 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
```
The last log message is repeatedly appearing.

### Affects Version(s)

1.8.0

### Kyuubi Server Log Output

```logtalk
: 26/01/25 04:24:11 INFO[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Processing g10_us's query[284c07fb-a061-4489-a725-e3d68b2174d9]: RUNNING_STATE -> ERROR_STATE, time taken: 7.146 seconds"

: 26/01/25 04:24:11 ERROR[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Exception in query[284c07fb-a061-4489-a725-e3d68b2174d9]: Error operating ExecuteStatement:[ org.apache.hadoop.mapred.InvalidInputException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.hadoop.mapred.InvalidInputException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124

: Caused by:[ java.io.IOException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Fjava.io.IOException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124

2026-01-25 04:24:13.942 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:18.943 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:23.944 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
```

### Kyuubi Engine Log Output

```logtalk

```

### Kyuubi Server Configurations

```yaml
spark.kyuubi.authentication.rest.bypass.url.pattern /api/v1/admin/engineMetrics
spark.kyuubi.backend.engine.exec.pool.size 600
spark.kyuubi.backend.engine.exec.pool.wait.queue.size 200
spark.kyuubi.backend.server.exec.pool.size 600
spark.kyuubi.backend.server.exec.pool.wait.queue.size 200
spark.kyuubi.client.ipAddress 7.36.133.145
spark.kyuubi.credentials.hadoopfs.uris hdfs://***
spark.kyuubi.engine.credentials
spark.kyuubi.engine.engineLog.path /home/hadoop/logs/kyuubi18/***
spark.kyuubi.engine.event.loggers CUSTOM,SPARK
spark.kyuubi.engine.metrics.internal.api.path /metrics/prometheus/sep
spark.kyuubi.engine.share.level.subdomain ***
spark.kyuubi.engine.submit.time 1769285944714
spark.kyuubi.engine.yarn.submit.timeout PT30M
spark.kyuubi.engineEnv.GDC_KYUUBI_USER g10_us
spark.kyuubi.frontend.protocols THRIFT_BINARY,REST
spark.kyuubi.frontend.thrift.max.worker.threads 5000
spark.kyuubi.frontend.thrift.min.worker.threads 200
spark.kyuubi.ha.addresses ***
spark.kyuubi.ha.enabled true
spark.kyuubi.ha.engine.lock.path /***
spark.kyuubi.ha.engine.ref.id db340152-1d8b-473d-b600-1053aa400dc8
spark.kyuubi.ha.namespace /***
spark.kyuubi.ha.zookeeper.auth.keytab ***
spark.kyuubi.ha.zookeeper.auth.principal ***
spark.kyuubi.ha.zookeeper.auth.type KERBEROS
spark.kyuubi.ha.zookeeper.client.port 2191
spark.kyuubi.ha.zookeeper.engine.auth.type KERBEROS
spark.kyuubi.ha.zookeeper.session.timeout 600000
spark.kyuubi.metrics.engine.prometheus.enable true
spark.kyuubi.metrics.engine.prometheus.path /engine_metrics/*
spark.kyuubi.metrics.host.enabled true
spark.kyuubi.metrics.prometheus.port 47070
spark.kyuubi.metrics.reporters PROMETHEUS
spark.kyuubi.server.ipAddress 10.90.133.157
spark.kyuubi.session.connection.url ***:37070
spark.kyuubi.session.engine.idle.timeout PT5M
spark.kyuubi.session.engine.initialize.timeout PT30M
spark.kyuubi.session.engine.spark.showProgress true
spark.kyuubi.session.idle.timeout PT180M
spark.kyuubi.session.real.user hive
spark.kyuubi.sql.engine.metrics.path /metrics/prometheus
```

### Kyuubi Engine Configurations

```yaml

```

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by inspecting org.apache.kyuubi.engine.spark.operation.ExecuteStatement and org.apache.kyuubi.operation.ExecuteStatement, focusing on the RUNNING_STATE to ERROR_STATE transition and the repeated KyuubiSessionManager-exec-pool messages. Reproduce the failure with the missing HDFS input path shown in the log, then verify that a failed query remains in ERROR_STATE and no longer continues polling until timeout or manual termination.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, scala, spark
Domain
backend, data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.