[Bug] Persistent metadata update failures can block batch session timeout cleanup
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched the existing open and closed issues and found no similar issue.
### Describe the bug
A persistent JDBC metadata update failure can remain in the asynchronous retry queue indefinitely and block batch session cleanup.
The relevant sequence is:
1. `JDBCMetadataStore.updateMetadata` throws when `PreparedStatement.executeUpdate()` returns `0`.
2. With `kyuubi.metadata.request.async.retry.enabled=true`, `MetadataManager.updateMetadata` enqueues the failed update.
3. The retry worker removes the queue head only after a successful update. A deterministic failure therefore remains at the head and is retried indefinitely.
4. `KyuubiBatchSession.close` waits without a timeout until the metadata retry queue is empty.
5. The session timeout checker is a single-thread scheduled executor and calls `closeSession` synchronously. One affected batch session can therefore prevent later sessions from being checked and removed. Old sessions may continue to appear in `/api/v1/sessions` even while their metadata rows are independently removed by the metadata cleaner.
In the observed sequence, a batch session was closed while its application monitor was still running. The monitor later observed the driver as `FAILED`, updated the application information, and the asynchronous operation's `finally` block attempted another terminal metadata update. This explains repeated terminal updates, but it has **not** been established that repetition alone caused Connector/J to return `0`.
The exact reason for the first zero update count is still under investigation. This report is about the deterministic failure amplification after any persistent zero-count update, not an assumption about its original cause.
#### Reproduction outline
1. Configure the MySQL JDBC metadata store with asynchronous retry enabled.
2. Submit a batch and make a terminal metadata update return `0`, for example by deleting the row before the update or by using a test metadata store that produces a deterministic zero-count update.
3. Close the corresponding batch session.
4. Observe that the same update remains at the retry queue head.
5. Observe that `KyuubiBatchSession.close` does not return and that the single session timeout checker cannot process later sessions.
#### Expected behavior
- Metadata persistence failures must not block session lifecycle cleanup indefinitely.
- A zero-count update should be reconciled using its postcondition instead of being treated as unconditional success or unconditional transient failure:
- if the row exists and all columns included in the update already match the intended values, the operation is idempotently complete;
- if the row is missing or its values do not match, retain a classified failure without silently recreating deleted metadata;
- preserve ordering when an earlier INSERT for the same identifier is pending.
- Persistent failures should use observable and bounded/rate-limited handling, and normal session close should not wait indefinitely for the asynchronous retry queue.
### Affects Version(s)
Observed on a deployment based on 1.9.1. The relevant code paths are also present on master at `804c00239815d3b7eece5c4c8bb8f6263063beff`.
### Kyuubi Server Log Output
```logtalk
INFO ... KyuubiBatchSession: Mark session SessionHandle [] closed
INFO ... BatchJobSubmission: Batch report for , Some(ApplicationInfo(...,FAILED,...))
ERROR ... MetadataManager: Error updating metadata for session
org.apache.kyuubi.KyuubiException: Error updating metadata for by SQL: UPDATE metadata SET ... WHERE identifier = ?
INFO ... MetadataManager: Retrying metadata requests for
ERROR ... MetadataManager: Error retrying metadata requests for
INFO ... KyuubiBatchSession: There are still remaining metadata store requests for batch[]
```
### Kyuubi Engine Log Output
Not required to reproduce the metadata retry and session-close behavior.
### Kyuubi Server Configurations
```yaml
kyuubi.metadata.store.jdbc.database.type: MYSQL
kyuubi.metadata.request.async.retry.enabled: true
kyuubi.metadata.request.retry.interval: 5s
```
The deployment is intended to use MySQL Connector/J 5.x and the JDBC URL does not explicitly set `useAffectedRows=true`; the effective runtime driver and connection properties are still being verified.
### Additional context
Relevant current-master code:
- [`JDBCMetadataStore.updateMetadata`](https://github.com/apache/kyuubi/blob/804c00239815d3b7eece5c4c8bb8f6263063beff/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala#L393-L401)
- [metadata asynchronous retry loop](https://github.com/apache/kyuubi/blob/804c00239815d3b7eece5c4c8bb8f6263063beff/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala#L322-L380)
- [`KyuubiBatchSession` waiting for retry completion](https://github.com/apache/kyuubi/blob/804c00239815d3b7eece5c4c8bb8f6263063beff/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiBatchSession.scala#L127-L136)
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix this.
Contributor guide
Research direction
Start with JDBCMetadataStore.scala, MetadataManager.scala, and KyuubiBatchSession.scala at the linked sections, then reproduce a deterministic zero-count update with asynchronous retry enabled. Trace the retry queue and session timeout behavior, including ordering for pending inserts. Done means persistent metadata failures are bounded and observable, zero-count updates are reconciled by postcondition, and session cleanup does not wait indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, scala
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100