apache / apache/kyuubi

[Bug] batch recovery crashes startup on null request_conf (KyuubiConf.set)

Open
#7,244 0 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

When Batch API v2 is enabled, Kyuubi persists batch requests’ conf map to the metadata store without null validation. On server restart, recovery loads these rows and constructs KyuubiBatchSession from metadata.request_conf. If any key has a null value (e.g., `"sfPassword": null`), `KyuubiConf.set` throws IllegalArgumentException (“value cannot be null for key: …”) during session construction in getBatchSessionsToRecover, which propagates and causes KyuubiRestFrontendService startup to fail.

**Expected**
the offending batch should be skipped/marked ERROR and startup should continue;

**Actual**
server startup aborts on the uncaught exception.

### Affects Version(s)

master

### Kyuubi Server Log Output

```logtalk
2025-11-15 02:53:58.945 ERROR org.apache.kyuubi.server.KyuubiServer: Error starting service KyuubiRestFrontendService
org.apache.kyuubi.KyuubiException: Cannot start KyuubiRestFrontendService
at org.apache.kyuubi.server.KyuubiRestFrontendService.start(KyuubiRestFrontendService.scala:239)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:53)
at org.apache.kyuubi.server.KyuubiServer.start(KyuubiServer.scala:225)
at org.apache.kyuubi.server.KyuubiServer$.startServer(KyuubiServer.scala:69)
at org.apache.kyuubi.server.KyuubiServer$.main(KyuubiServer.scala:104)
at org.apache.kyuubi.server.KyuubiServer.main(KyuubiServer.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: value cannot be null for key: sfPassword
at scala.Predef$.require(Predef.scala:281)
at org.apache.kyuubi.config.KyuubiConf.set(KyuubiConf.scala:79)
at org.apache.kyuubi.session.KyuubiBatchSession.$anonfun$new$1(KyuubiBatchSession.scala:102)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:400)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at org.apache.kyuubi.session.KyuubiBatchSession.(KyuubiBatchSession.scala:101)
at org.apache.kyuubi.session.KyuubiSessionManager.createBatchSession(KyuubiSessionManager.scala:174)
at org.apache.kyuubi.session.KyuubiSessionManager.$anonfun$getBatchSessionsToRecover$3(KyuubiSessionManager.scala:326)
at scala.collection.immutable.List.map(List.scala:293)
at org.apache.kyuubi.session.KyuubiSessionManager.$anonfun$getBatchSessionsToRecover$2(KyuubiSessionManager.scala:314)
at scala.Option.map(Option.scala:230)
at org.apache.kyuubi.session.KyuubiSessionManager.$anonfun$getBatchSessionsToRecover$1(KyuubiSessionManager.scala:314)
at scala.collection.immutable.List.flatMap(List.scala:366)
at org.apache.kyuubi.session.KyuubiSessionManager.getBatchSessionsToRecover(KyuubiSessionManager.scala:309)
at org.apache.kyuubi.server.KyuubiRestFrontendService.recoverBatchSessions(KyuubiRestFrontendService.scala:176)
at org.apache.kyuubi.server.KyuubiRestFrontendService.start(KyuubiRestFrontendService.scala:234)
... 11 more
2025-11-15 02:53:58.947 INFO org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiBackendService] is stopping.
2025-11-15 02:53:58.947 INFO org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiSessionManager] is stopping.
2025-11-15 02:53:58.947 INFO org.apache.kyuubi.session.KyuubiSessionManager: Service: [KyuubiOperationManager] is stopping.
2025-11-15 02:53:58.948 INFO org.apache.kyuubi.operation.KyuubiOperationManager: Service[KyuubiOperationManager] is stopped.
2025-11-15 02:53:58.948 INFO org.apache.kyuubi.session.KyuubiSessionManager: Service: [MetadataManager] is stopping.
2025-11-15 02:53:58.949 INFO com.zaxxer.hikari.HikariDataSource: jdbc-metadata-store-pool - Shutdown initiated...
2025-11-15 02:53:58.949 DEBUG com.zaxxer.hikari.pool.HikariPool: jdbc-metadata-store-pool - Before shutdown stats (total=9, active=0, idle=9, waiting=0)
2025-11-15 02:53:58.978 DEBUG com.zaxxer.hikari.pool.HikariPool: jdbc-metadata-store-pool - Added connection org.postgresql.jdbc.PgConnection@7689f3b5
2025-11-15 02:53:59.073 DEBUG com.zaxxer.hikari.pool.HikariPool: jdbc-metadata-store-pool - After shutdown stats (total=0, active=0, idle=0, waiting=0)
2025-11-15 02:53:59.073 INFO com.zaxxer.hikari.HikariDataSource: jdbc-metadata-store-pool - Shutdown completed.
2025-11-15 02:53:59.074 INFO org.apache.kyuubi.server.metadata.MetadataManager: Service[MetadataManager] is stopped.
```

### Kyuubi Engine Log Output

```logtalk
N/A
```

### Kyuubi Server Configurations

```yaml
kyuubi.session.engine.startup.waitCompletion=false
kyuubi.batch.impl.version=2
kyuubi.batch.submitter.enabled=true
kyuubi.batch.submitter.threads=10
```

### Kyuubi Engine Configurations

```yaml
N/A
```

### Additional context

_No response_

### Are you willing to submit PR?

- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start with KyuubiSessionManager.scala at getBatchSessionsToRecover and createBatchSession, then trace KyuubiBatchSession.scala and KyuubiConf.scala where the stack trace identifies the failure. Check how KyuubiRestFrontendService.scala recovers batches and how request_conf is loaded. Done means a null-valued request configuration no longer aborts startup, and the offending batch is skipped or marked ERROR.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.