apache / apache/kyuubi

[Bug] Intermittent OperationHandle 404 when fetching results under concurrent load

Open
#7,363 2 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

I am performing a concurrency stress test on Kyuubi and encountered intermittent 404 errors when fetching operation results.

The issue appears to occur when calling get result row set for an operation that has already transitioned to CLOSED_STATE, even though the client did not explicitly close the operation.

**Test Setup**
Goal

Validate the maximum concurrent Operation capacity supported by 4 Kyuubi nodes under the same environment configuration.

Load Testing Method

Using Locust to simulate client behavior with the following workflow:

Create operation

Poll operation status

Fetch operation result

Repeat continuously

Test Parameters

Concurrency: 60

Duration: 10 minutes continuous execution

**Observed Behavior**

Intermittent failures when fetching results:

`Error getting result row set for operation handle
`
Example error:

`org.apache.kyuubi.KyuubiSQLException: Invalid OperationHandle [a0f5cbc3-08a4-4a54-8e38-e944174fa940]
`
This corresponds to a 404-like scenario where the operation handle is no longer valid.

### Affects Version(s)

1.9.3

### Kyuubi Server Log Output

```logtalk
From Kyuubi server logs, the lifecycle of a failed operation is as follows:
2026-03-20 17:36:33.524 INFO OperationLog - Creating operation log file ...

2026-03-20 17:42:53.342 INFO ExecuteStatement - PENDING_STATE -> RUNNING_STATE

2026-03-20 17:42:54.572 INFO ExecuteStatement - RUNNING_STATE -> FINISHED_STATE

2026-03-20 17:44:42.814 INFO ExecuteStatement - Processing zhou's query[a0f5cbc3-08a4-4a54-8e38-e944174fa940]: FINISHED_STATE -> CLOSED_STATE, time taken: 109.472 seconds

2026-03-20 17:44:42.814 ERROR ApiUtils - Error getting result row set for operation handle a0f5cbc3-08a4-4a54-8e38-e944174fa940
org.apache.kyuubi.KyuubiSQLException: Invalid OperationHandle [a0f5cbc3-08a4-4a54-8e38-e944174fa940]
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.operation.OperationManager.getOperation(OperationManager.scala:106)
at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$fetchResults$1(BackendServiceMetric.scala:211)
at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:112)
at org.apache.kyuubi.server.BackendServiceMetric.fetchResults(BackendServiceMetric.scala:186)
at org.apache.kyuubi.server.BackendServiceMetric.fetchResults$(BackendServiceMetric.scala:181)
at org.apache.kyuubi.server.KyuubiServer$$anon$1.fetchResults(KyuubiServer.scala:179)
at org.apache.kyuubi.server.api.v1.OperationsResource.getNextRowSet(OperationsResource.scala:186)
```

Additionally, I noticed that before the error occurs, there are usually logs indicating that the ZooKeeper session was closed. I’m not sure whether this is related to the issue.

Also, in the Kyuubi server logs, there are a large number of entries showing frequent connections being established to ZooKeeper.
Image

### Kyuubi Engine Log Output

```logtalk

```

### Kyuubi Server Configurations

```yaml

```

### Kyuubi Engine Configurations

```yaml

```

### Additional context

Not all operations are affected。

Image

Image

Image

Image

Image

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start with the reported stack trace in operation/OperationManager.scala, server/BackendServiceMetric.scala, and server/api/v1/OperationsResource.scala, then trace the operation lifecycle from FINISHED_STATE to CLOSED_STATE. Reproduce the Locust workflow under concurrent load and inspect the ZooKeeper session-related logs. Done means result fetching no longer intermittently rejects a valid operation handle in this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.