[Bug] batch recovery crashes startup on null request_conf (KyuubiConf.set)
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
When Batch API v2 is enabled, Kyuubi persists batch requests’ conf map to the metadata store without null validation. On server restart, recovery loads these rows and constructs KyuubiBatchSession from metadata.request_conf. If any key has a null value (e.g., `"sfPassword": null`), `KyuubiConf.set` throws IllegalArgumentException (“value cannot be null for key: …”) during session construction in getBatchSessionsToRecover, which propagates and causes KyuubiRestFrontendService startup to fail.
**Expected**
the offending batch should be skipped/marked ERROR and startup should continue;
**Actual**
server startup aborts on the uncaught exception.
### Affects Version(s)
master
### Kyuubi Server Log Output
```logtalk
2025-11-15 02:53:58.945 ERROR org.apache.kyuubi.server.KyuubiServer: Error starting service KyuubiRestFrontendService
org.apache.kyuubi.KyuubiException: Cannot start KyuubiRestFrontendService
at org.apache.kyuubi.server.KyuubiRestFrontendService.start(KyuubiRestFrontendService.scala:239)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:53)
at org.apache.kyuubi.server.KyuubiServer.start(KyuubiServer.scala:225)
at org.apache.kyuubi.server.KyuubiServer$.startServer(KyuubiServer.scala:69)
at org.apache.kyuubi.server.KyuubiServer$.main(KyuubiServer.scala:104)
at org.apache.kyuubi.server.KyuubiServer.main(KyuubiServer.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: value cannot be null for key: sfPassword
at scala.Predef$.require(Predef.scala:281)
at org.apache.kyuubi.config.KyuubiConf.set(KyuubiConf.scala:79)
at org.apache.kyuubi.session.KyuubiBatchSession.$anonfun$new$1(KyuubiBatchSession.scala:102)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:400)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at org.apache.kyuubi.session.KyuubiBatchSession.(KyuubiBatchSession.scala:101)
at org.apache.kyuubi.session.KyuubiSessionManager.createBatchSession(KyuubiSessionManager.scala:174)
at org.apache.kyuubi.session.KyuubiSessionManager.$anonfun$getBatchSessionsToRecover$3(KyuubiSessionManager.scala:326)
at scala.collection.immutable.List.map(List.scala:293)
at org.apache.kyuubi.session.KyuubiSessionManager.$anonfun$getBatchSessionsToRecover$2(KyuubiSessionManager.scala:314)
at scala.Option.map(Option.scala:230)
at org.apache.kyuubi.session.KyuubiSessionManager.$anonfun$getBatchSessionsToRecover$1(KyuubiSessionManager.scala:314)
at scala.collection.immutable.List.flatMap(List.scala:366)
at org.apache.kyuubi.session.KyuubiSessionManager.getBatchSessionsToRecover(KyuubiSessionManager.scala:309)
at org.apache.kyuubi.server.KyuubiRestFrontendService.recoverBatchSessions(KyuubiRestFrontendService.scala:176)
at org.apache.kyuubi.server.KyuubiRestFrontendService.start(KyuubiRestFrontendService.scala:234)
... 11 more
2025-11-15 02:53:58.947 INFO org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiBackendService] is stopping.
2025-11-15 02:53:58.947 INFO org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiSessionManager] is stopping.
2025-11-15 02:53:58.947 INFO org.apache.kyuubi.session.KyuubiSessionManager: Service: [KyuubiOperationManager] is stopping.
2025-11-15 02:53:58.948 INFO org.apache.kyuubi.operation.KyuubiOperationManager: Service[KyuubiOperationManager] is stopped.
2025-11-15 02:53:58.948 INFO org.apache.kyuubi.session.KyuubiSessionManager: Service: [MetadataManager] is stopping.
2025-11-15 02:53:58.949 INFO com.zaxxer.hikari.HikariDataSource: jdbc-metadata-store-pool - Shutdown initiated...
2025-11-15 02:53:58.949 DEBUG com.zaxxer.hikari.pool.HikariPool: jdbc-metadata-store-pool - Before shutdown stats (total=9, active=0, idle=9, waiting=0)
2025-11-15 02:53:58.978 DEBUG com.zaxxer.hikari.pool.HikariPool: jdbc-metadata-store-pool - Added connection org.postgresql.jdbc.PgConnection@7689f3b5
2025-11-15 02:53:59.073 DEBUG com.zaxxer.hikari.pool.HikariPool: jdbc-metadata-store-pool - After shutdown stats (total=0, active=0, idle=0, waiting=0)
2025-11-15 02:53:59.073 INFO com.zaxxer.hikari.HikariDataSource: jdbc-metadata-store-pool - Shutdown completed.
2025-11-15 02:53:59.074 INFO org.apache.kyuubi.server.metadata.MetadataManager: Service[MetadataManager] is stopped.
```
### Kyuubi Engine Log Output
```logtalk
N/A
```
### Kyuubi Server Configurations
```yaml
kyuubi.session.engine.startup.waitCompletion=false
kyuubi.batch.impl.version=2
kyuubi.batch.submitter.enabled=true
kyuubi.batch.submitter.threads=10
```
### Kyuubi Engine Configurations
```yaml
N/A
```
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Start with KyuubiSessionManager.scala at getBatchSessionsToRecover and createBatchSession, then trace KyuubiBatchSession.scala and KyuubiConf.scala where the stack trace identifies the failure. Check how KyuubiRestFrontendService.scala recovers batches and how request_conf is loaded. Done means a null-valued request configuration no longer aborts startup, and the offending batch is skipped or marked ERROR.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100