[Bug] The Kyuubi query did not take effect when entering the ERROR_STATE from RUNNING-STATE
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
The Kyuubi query did not take effect when entering the ERROR_STATE from RUNNING-STATE, and eventually changed back to RUNNING-STATE. Then keep running continuously until timeout or manual kill. The log show below.
```
: 26/01/25 04:24:11 INFO[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Processing g10_us's query[284c07fb-a061-4489-a725-e3d68b2174d9]: RUNNING_STATE -> ERROR_STATE, time taken: 7.146 seconds"
: 26/01/25 04:24:11 ERROR[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Exception in query[284c07fb-a061-4489-a725-e3d68b2174d9]: Error operating ExecuteStatement:[ org.apache.hadoop.mapred.InvalidInputException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.hadoop.mapred.InvalidInputException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124
: Caused by:[ java.io.IOException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Fjava.io.IOException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124
2026-01-25 04:24:13.942 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:18.943 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:23.944 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
```
The last log message is repeatedly appearing.
### Affects Version(s)
1.8.0
### Kyuubi Server Log Output
```logtalk
: 26/01/25 04:24:11 INFO[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Processing g10_us's query[284c07fb-a061-4489-a725-e3d68b2174d9]: RUNNING_STATE -> ERROR_STATE, time taken: 7.146 seconds"
: 26/01/25 04:24:11 ERROR[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Exception in query[284c07fb-a061-4489-a725-e3d68b2174d9]: Error operating ExecuteStatement:[ org.apache.hadoop.mapred.InvalidInputException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.hadoop.mapred.InvalidInputException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124
: Caused by:[ java.io.IOException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Fjava.io.IOException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124
2026-01-25 04:24:13.942 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:18.943 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:23.944 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
```
### Kyuubi Engine Log Output
```logtalk
```
### Kyuubi Server Configurations
```yaml
spark.kyuubi.authentication.rest.bypass.url.pattern /api/v1/admin/engineMetrics
spark.kyuubi.backend.engine.exec.pool.size 600
spark.kyuubi.backend.engine.exec.pool.wait.queue.size 200
spark.kyuubi.backend.server.exec.pool.size 600
spark.kyuubi.backend.server.exec.pool.wait.queue.size 200
spark.kyuubi.client.ipAddress 7.36.133.145
spark.kyuubi.credentials.hadoopfs.uris hdfs://***
spark.kyuubi.engine.credentials
spark.kyuubi.engine.engineLog.path /home/hadoop/logs/kyuubi18/***
spark.kyuubi.engine.event.loggers CUSTOM,SPARK
spark.kyuubi.engine.metrics.internal.api.path /metrics/prometheus/sep
spark.kyuubi.engine.share.level.subdomain ***
spark.kyuubi.engine.submit.time 1769285944714
spark.kyuubi.engine.yarn.submit.timeout PT30M
spark.kyuubi.engineEnv.GDC_KYUUBI_USER g10_us
spark.kyuubi.frontend.protocols THRIFT_BINARY,REST
spark.kyuubi.frontend.thrift.max.worker.threads 5000
spark.kyuubi.frontend.thrift.min.worker.threads 200
spark.kyuubi.ha.addresses ***
spark.kyuubi.ha.enabled true
spark.kyuubi.ha.engine.lock.path /***
spark.kyuubi.ha.engine.ref.id db340152-1d8b-473d-b600-1053aa400dc8
spark.kyuubi.ha.namespace /***
spark.kyuubi.ha.zookeeper.auth.keytab ***
spark.kyuubi.ha.zookeeper.auth.principal ***
spark.kyuubi.ha.zookeeper.auth.type KERBEROS
spark.kyuubi.ha.zookeeper.client.port 2191
spark.kyuubi.ha.zookeeper.engine.auth.type KERBEROS
spark.kyuubi.ha.zookeeper.session.timeout 600000
spark.kyuubi.metrics.engine.prometheus.enable true
spark.kyuubi.metrics.engine.prometheus.path /engine_metrics/*
spark.kyuubi.metrics.host.enabled true
spark.kyuubi.metrics.prometheus.port 47070
spark.kyuubi.metrics.reporters PROMETHEUS
spark.kyuubi.server.ipAddress 10.90.133.157
spark.kyuubi.session.connection.url ***:37070
spark.kyuubi.session.engine.idle.timeout PT5M
spark.kyuubi.session.engine.initialize.timeout PT30M
spark.kyuubi.session.engine.spark.showProgress true
spark.kyuubi.session.idle.timeout PT180M
spark.kyuubi.session.real.user hive
spark.kyuubi.sql.engine.metrics.path /metrics/prometheus
```
### Kyuubi Engine Configurations
```yaml
```
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Start by inspecting org.apache.kyuubi.engine.spark.operation.ExecuteStatement and org.apache.kyuubi.operation.ExecuteStatement, focusing on the RUNNING_STATE to ERROR_STATE transition and the repeated KyuubiSessionManager-exec-pool messages. Reproduce the failure with the missing HDFS input path shown in the log, then verify that a failed query remains in ERROR_STATE and no longer continues polling until timeout or manual termination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, scala, spark
- Domain
- backend, data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100