apache / apache/kyuubi

[Bug] Persistent metadata update failures can block batch session timeout cleanup

Open
#7,708 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched the existing open and closed issues and found no similar issue.

### Describe the bug

A persistent JDBC metadata update failure can remain in the asynchronous retry queue indefinitely and block batch session cleanup.

The relevant sequence is:

1. `JDBCMetadataStore.updateMetadata` throws when `PreparedStatement.executeUpdate()` returns `0`.
2. With `kyuubi.metadata.request.async.retry.enabled=true`, `MetadataManager.updateMetadata` enqueues the failed update.
3. The retry worker removes the queue head only after a successful update. A deterministic failure therefore remains at the head and is retried indefinitely.
4. `KyuubiBatchSession.close` waits without a timeout until the metadata retry queue is empty.
5. The session timeout checker is a single-thread scheduled executor and calls `closeSession` synchronously. One affected batch session can therefore prevent later sessions from being checked and removed. Old sessions may continue to appear in `/api/v1/sessions` even while their metadata rows are independently removed by the metadata cleaner.

In the observed sequence, a batch session was closed while its application monitor was still running. The monitor later observed the driver as `FAILED`, updated the application information, and the asynchronous operation's `finally` block attempted another terminal metadata update. This explains repeated terminal updates, but it has **not** been established that repetition alone caused Connector/J to return `0`.

The exact reason for the first zero update count is still under investigation. This report is about the deterministic failure amplification after any persistent zero-count update, not an assumption about its original cause.

#### Reproduction outline

1. Configure the MySQL JDBC metadata store with asynchronous retry enabled.
2. Submit a batch and make a terminal metadata update return `0`, for example by deleting the row before the update or by using a test metadata store that produces a deterministic zero-count update.
3. Close the corresponding batch session.
4. Observe that the same update remains at the retry queue head.
5. Observe that `KyuubiBatchSession.close` does not return and that the single session timeout checker cannot process later sessions.

#### Expected behavior

- Metadata persistence failures must not block session lifecycle cleanup indefinitely.
- A zero-count update should be reconciled using its postcondition instead of being treated as unconditional success or unconditional transient failure:
- if the row exists and all columns included in the update already match the intended values, the operation is idempotently complete;
- if the row is missing or its values do not match, retain a classified failure without silently recreating deleted metadata;
- preserve ordering when an earlier INSERT for the same identifier is pending.
- Persistent failures should use observable and bounded/rate-limited handling, and normal session close should not wait indefinitely for the asynchronous retry queue.

### Affects Version(s)

Observed on a deployment based on 1.9.1. The relevant code paths are also present on master at `804c00239815d3b7eece5c4c8bb8f6263063beff`.

### Kyuubi Server Log Output

```logtalk
INFO ... KyuubiBatchSession: Mark session SessionHandle [] closed
INFO ... BatchJobSubmission: Batch report for , Some(ApplicationInfo(...,FAILED,...))
ERROR ... MetadataManager: Error updating metadata for session
org.apache.kyuubi.KyuubiException: Error updating metadata for by SQL: UPDATE metadata SET ... WHERE identifier = ?
INFO ... MetadataManager: Retrying metadata requests for
ERROR ... MetadataManager: Error retrying metadata requests for
INFO ... KyuubiBatchSession: There are still remaining metadata store requests for batch[]
```

### Kyuubi Engine Log Output

Not required to reproduce the metadata retry and session-close behavior.

### Kyuubi Server Configurations

```yaml
kyuubi.metadata.store.jdbc.database.type: MYSQL
kyuubi.metadata.request.async.retry.enabled: true
kyuubi.metadata.request.retry.interval: 5s
```

The deployment is intended to use MySQL Connector/J 5.x and the JDBC URL does not explicitly set `useAffectedRows=true`; the effective runtime driver and connection properties are still being verified.

### Additional context

Relevant current-master code:

- [`JDBCMetadataStore.updateMetadata`](https://github.com/apache/kyuubi/blob/804c00239815d3b7eece5c4c8bb8f6263063beff/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/jdbc/JDBCMetadataStore.scala#L393-L401)
- [metadata asynchronous retry loop](https://github.com/apache/kyuubi/blob/804c00239815d3b7eece5c4c8bb8f6263063beff/kyuubi-server/src/main/scala/org/apache/kyuubi/server/metadata/MetadataManager.scala#L322-L380)
- [`KyuubiBatchSession` waiting for retry completion](https://github.com/apache/kyuubi/blob/804c00239815d3b7eece5c4c8bb8f6263063beff/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiBatchSession.scala#L127-L136)

### Are you willing to submit PR?

- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix this.

Contributor guide

Open the contributing guide

Research direction

Start with JDBCMetadataStore.scala, MetadataManager.scala, and KyuubiBatchSession.scala at the linked sections, then reproduce a deterministic zero-count update with asynchronous retry enabled. Trace the retry queue and session timeout behavior, including ordering for pending inserts. Done means persistent metadata failures are bounded and observable, zero-count updates are reconciled by postcondition, and session cleanup does not wait indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, scala
Domain
backend, database
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.