[Bug] Prepared INSERT forwarded from a non-master FE fails with NoSuchElementException (errCode 1105) on cached COM_STMT_EXECUTE
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I searched the [issues](https://github.com/apache/doris/issues) and found no similar issue.
### Version
Apache Doris **4.1.1** (docker image `dyrnq/doris:4.1.1`).
Also reproduced identically on **4.1.0** and **4.1.2** — not a regression within 4.1.x.
### What's Wrong?
When a client uses the MySQL **binary prepared-statement protocol** with statement caching (i.e. it omits the parameter TYPE bytes on repeat executes — `new-params-bound-flag = 0`) and sends a prepared **write** (`INSERT`) to a **non-master (follower) FE**, the execute is forwarded to the master FE via `FrontendServiceImpl.forward`. On the master, the forwarded execute has **no placeholder types**, so `Placeholder.getMysqlColType()` calls `Optional.get()` on an empty `Optional` and throws:
```
java.util.NoSuchElementException: No value present
at java.base/java.util.Optional.get(Optional.java:143)
at org.apache.doris.nereids.trees.expressions.Placeholder.getMysqlColType(Placeholder.java:97)
at org.apache.doris.qe.MysqlConnectProcessor.handleExecute(MysqlConnectProcessor.java:156)
at org.apache.doris.qe.ConnectProcessor.proxyExecute(ConnectProcessor.java:753)
at org.apache.doris.service.FrontendServiceImpl.forward(FrontendServiceImpl.java:1137)
at ...
```
The client sees: `{errorMessage=NoSuchElementException, msg: No value present, errorCode=1105, sqlState=HY000}`.
Key characteristics:
- **Per-execute, not per-connection.** The **first** execute on a connection carries the placeholder type bytes and forwards fine; every **cached** repeat execute omits them and fails. In a 30-execute loop on one connection we consistently get `OK=1, failures=29`.
- **Master vs follower.** The exact same client and loop against the **master** FE succeeds 30/30. Only executes **forwarded from a follower** fail. This shows Doris handles `new-params-bound-flag = 0` correctly when the FE that ran `COM_STMT_PREPARE` also runs the execute — it only breaks across the forward.
- **The write never happens** — it fails on the master before planning/execution.
This is deployment-mode-independent (reproduced in coupled/local mode); the defect is in the FE prepared-statement forward path, not in storage.
### What You Expected?
A prepared `INSERT` sent to a follower FE should behave exactly like one sent to the master — the forwarded execute should carry (or the master should recover) the placeholder types, and the write should succeed. Omitting parameter types on repeat executes is a legal MySQL binary-protocol optimization, so the master's forward path must tolerate it. At the very least, the server should never answer a legal protocol packet with an internal `NoSuchElementException` from an unguarded `Optional.get()` — it should return a clean, retryable error.
The current behavior makes prepared writes fail ~non-deterministically on any multi-FE cluster (rate depends on how the connection pool mixes fresh vs cached executes and how it routes across FEs).
### How to Reproduce?
**1. A cluster with a follower FE** (1 master FE + 1 follower FE + 1 BE). Any topology where the client can connect to a non-master FE works.
**2. A minimal table:**
```sql
CREATE DATABASE IF NOT EXISTS repro;
CREATE TABLE IF NOT EXISTS repro.t (
k INT NOT NULL,
v VARCHAR(50)
)
DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES ("replication_num" = "1");
```
**3. A client that omits parameter types on cached executes.** The vert.x reactive MySQL client with `setCachePreparedStatements(true)` does this (it caches the prepared statement per connection and sends `new-params-bound-flag = 0` on repeat executes for the same SQL):
```java
// io.vertx:vertx-mysql-client:4.5.11
MySQLConnectOptions opts = new MySQLConnectOptions()
.setHost(host).setPort(FOLLOWER_QUERY_PORT).setUser("root").setDatabase("repro")
.setCachePreparedStatements(true).setSslMode(SslMode.DISABLED);
SqlConnection conn = MySQLConnection.connect(vertx, opts).toCompletionStage().toCompletableFuture().get();
for (int i = 0; i < 30; i++) {
// First iteration sends placeholder types -> forwarded OK.
// Every later iteration reuses the cached statement and omits types -> forwarded execute fails on master.
conn.preparedQuery("INSERT INTO repro.t (k, v) VALUES (?, ?)")
.execute(Tuple.of(i, "v" + i))
.toCompletionStage().toCompletableFuture().get();
}
```
**Result:**
| Target FE | Outcome |
|---|---|
| **Master** | 30/30 succeed |
| **Follower** | **1/30 succeed, 29/30 fail** with `NoSuchElementException` / errCode 1105 (the stack above appears in the master FE's `fe.log`) |
**Mitigation that confirms the mechanism:** using a **fresh connection per insert** (so every execute is a "first execute" that carries types) → **0 failures** against the follower.
### Anything Else?
**Client dependency (important for triage).** The bug only manifests with clients that omit the parameter TYPE bytes on cached executes. We verified against the same follower:
| Client | Behavior on repeat execute | Reproduces? |
|---|---|---|
| vert.x mysql-client 4.5.11 (`cachePreparedStatements=true`) | omits types (`new-params-bound-flag=0`) | **Yes** |
| MySQL Connector/J 8.4 (`useServerPrepStmts=true&cachePrepStmts=true`), single prepared execute | re-sends types every execute | No |
| MySQL Connector/J 8.4, server-prepared **batch** (`rewriteBatchedStatements=false`) | re-sends types every execute | No |
| mysql-connector-python (`prepared=True`) | re-sends types every execute | No |
So the trigger is the standard, spec-legal MySQL optimization of not re-sending types when they haven't changed. The clients that re-send types every execute happen to mask it. The master's forward path should tolerate the type-omitted case.
**Likely origin.** The non-master prepared-statement forwarding path appears to come from the feature that added prepared-statement forwarding to the master (#48689, built on Nereids prepared statements in #35318). #63920 (present in 4.1.2) touches the prepared-statement path but is point-query(read)-only and does **not** cover this forwarded-INSERT case (verified: 4.1.2 still reproduces). Root cause looks like `MysqlConnectProcessor.handleExecute` / `ConnectProcessor.proxyExecute` not carrying the placeholder MySQL column types across the `forward` RPC, so on the master `Placeholder.getMysqlColType()` finds an empty `Optional`.
**Suggested fix directions:** either (a) carry the placeholder types in the forwarded request so the master can rebuild them, or (b) have the master re-derive types from the bound values / fall back gracefully instead of `Optional.get()`, or at minimum (c) return a clear, retryable error instead of an internal `NoSuchElementException`.
Contributor guide
Research direction
Start with Placeholder.java, MysqlConnectProcessor.java, ConnectProcessor.java, and FrontendServiceImpl.java, tracing the cached COM_STMT_EXECUTE path through forwarding. Reproduce with the Vert.x MySQL client loop against a follower, then verify that prepared INSERTs with omitted parameter types succeed or return a clean retryable error instead of NoSuchElementException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100