apache / apache/doris

[Bug] Prepared INSERT forwarded from a non-master FE fails with NoSuchElementException (errCode 1105) on cached COM_STMT_EXECUTE

Open
#65,848 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I searched the [issues](https://github.com/apache/doris/issues) and found no similar issue.

### Version

Apache Doris **4.1.1** (docker image `dyrnq/doris:4.1.1`).
Also reproduced identically on **4.1.0** and **4.1.2** — not a regression within 4.1.x.

### What's Wrong?

When a client uses the MySQL **binary prepared-statement protocol** with statement caching (i.e. it omits the parameter TYPE bytes on repeat executes — `new-params-bound-flag = 0`) and sends a prepared **write** (`INSERT`) to a **non-master (follower) FE**, the execute is forwarded to the master FE via `FrontendServiceImpl.forward`. On the master, the forwarded execute has **no placeholder types**, so `Placeholder.getMysqlColType()` calls `Optional.get()` on an empty `Optional` and throws:

```
java.util.NoSuchElementException: No value present
at java.base/java.util.Optional.get(Optional.java:143)
at org.apache.doris.nereids.trees.expressions.Placeholder.getMysqlColType(Placeholder.java:97)
at org.apache.doris.qe.MysqlConnectProcessor.handleExecute(MysqlConnectProcessor.java:156)
at org.apache.doris.qe.ConnectProcessor.proxyExecute(ConnectProcessor.java:753)
at org.apache.doris.service.FrontendServiceImpl.forward(FrontendServiceImpl.java:1137)
at ...
```

The client sees: `{errorMessage=NoSuchElementException, msg: No value present, errorCode=1105, sqlState=HY000}`.

Key characteristics:

- **Per-execute, not per-connection.** The **first** execute on a connection carries the placeholder type bytes and forwards fine; every **cached** repeat execute omits them and fails. In a 30-execute loop on one connection we consistently get `OK=1, failures=29`.
- **Master vs follower.** The exact same client and loop against the **master** FE succeeds 30/30. Only executes **forwarded from a follower** fail. This shows Doris handles `new-params-bound-flag = 0` correctly when the FE that ran `COM_STMT_PREPARE` also runs the execute — it only breaks across the forward.
- **The write never happens** — it fails on the master before planning/execution.

This is deployment-mode-independent (reproduced in coupled/local mode); the defect is in the FE prepared-statement forward path, not in storage.

### What You Expected?

A prepared `INSERT` sent to a follower FE should behave exactly like one sent to the master — the forwarded execute should carry (or the master should recover) the placeholder types, and the write should succeed. Omitting parameter types on repeat executes is a legal MySQL binary-protocol optimization, so the master's forward path must tolerate it. At the very least, the server should never answer a legal protocol packet with an internal `NoSuchElementException` from an unguarded `Optional.get()` — it should return a clean, retryable error.

The current behavior makes prepared writes fail ~non-deterministically on any multi-FE cluster (rate depends on how the connection pool mixes fresh vs cached executes and how it routes across FEs).

### How to Reproduce?

**1. A cluster with a follower FE** (1 master FE + 1 follower FE + 1 BE). Any topology where the client can connect to a non-master FE works.

**2. A minimal table:**

```sql
CREATE DATABASE IF NOT EXISTS repro;
CREATE TABLE IF NOT EXISTS repro.t (
k INT NOT NULL,
v VARCHAR(50)
)
DUPLICATE KEY(k)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES ("replication_num" = "1");
```

**3. A client that omits parameter types on cached executes.** The vert.x reactive MySQL client with `setCachePreparedStatements(true)` does this (it caches the prepared statement per connection and sends `new-params-bound-flag = 0` on repeat executes for the same SQL):

```java
// io.vertx:vertx-mysql-client:4.5.11
MySQLConnectOptions opts = new MySQLConnectOptions()
.setHost(host).setPort(FOLLOWER_QUERY_PORT).setUser("root").setDatabase("repro")
.setCachePreparedStatements(true).setSslMode(SslMode.DISABLED);
SqlConnection conn = MySQLConnection.connect(vertx, opts).toCompletionStage().toCompletableFuture().get();

for (int i = 0; i < 30; i++) {
// First iteration sends placeholder types -> forwarded OK.
// Every later iteration reuses the cached statement and omits types -> forwarded execute fails on master.
conn.preparedQuery("INSERT INTO repro.t (k, v) VALUES (?, ?)")
.execute(Tuple.of(i, "v" + i))
.toCompletionStage().toCompletableFuture().get();
}
```

**Result:**

| Target FE | Outcome |
|---|---|
| **Master** | 30/30 succeed |
| **Follower** | **1/30 succeed, 29/30 fail** with `NoSuchElementException` / errCode 1105 (the stack above appears in the master FE's `fe.log`) |

**Mitigation that confirms the mechanism:** using a **fresh connection per insert** (so every execute is a "first execute" that carries types) → **0 failures** against the follower.

### Anything Else?

**Client dependency (important for triage).** The bug only manifests with clients that omit the parameter TYPE bytes on cached executes. We verified against the same follower:

| Client | Behavior on repeat execute | Reproduces? |
|---|---|---|
| vert.x mysql-client 4.5.11 (`cachePreparedStatements=true`) | omits types (`new-params-bound-flag=0`) | **Yes** |
| MySQL Connector/J 8.4 (`useServerPrepStmts=true&cachePrepStmts=true`), single prepared execute | re-sends types every execute | No |
| MySQL Connector/J 8.4, server-prepared **batch** (`rewriteBatchedStatements=false`) | re-sends types every execute | No |
| mysql-connector-python (`prepared=True`) | re-sends types every execute | No |

So the trigger is the standard, spec-legal MySQL optimization of not re-sending types when they haven't changed. The clients that re-send types every execute happen to mask it. The master's forward path should tolerate the type-omitted case.

**Likely origin.** The non-master prepared-statement forwarding path appears to come from the feature that added prepared-statement forwarding to the master (#48689, built on Nereids prepared statements in #35318). #63920 (present in 4.1.2) touches the prepared-statement path but is point-query(read)-only and does **not** cover this forwarded-INSERT case (verified: 4.1.2 still reproduces). Root cause looks like `MysqlConnectProcessor.handleExecute` / `ConnectProcessor.proxyExecute` not carrying the placeholder MySQL column types across the `forward` RPC, so on the master `Placeholder.getMysqlColType()` finds an empty `Optional`.

**Suggested fix directions:** either (a) carry the placeholder types in the forwarded request so the master can rebuild them, or (b) have the master re-derive types from the bound values / fall back gracefully instead of `Optional.get()`, or at minimum (c) return a clear, retryable error instead of an internal `NoSuchElementException`.

Contributor guide

Open the contributing guide

Research direction

Start with Placeholder.java, MysqlConnectProcessor.java, ConnectProcessor.java, and FrontendServiceImpl.java, tracing the cached COM_STMT_EXECUTE path through forwarding. Reproduce with the Vert.x MySQL client loop against a follower, then verify that prepared INSERTs with omitted parameter types succeed or return a clean retryable error instead of NoSuchElementException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.