apache / apache/doris

[Bug] Observer/follower keeps forwarding statements to old (degraded) master after master failover, clients receive errors for minutes

Open
#67,297 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues.

### Version

Doris 3.0.8 (also verified against master branch code)

### What's Wrong?

After a master FE failover (old master lost leadership due to OOM/GC pause but the process stayed alive and kept serving its MySQL/thrift ports), a non-master FE (observer/follower) with a lagging journal replay keeps forwarding statements to the **old master**, and clients continuously receive:

```
ERROR 1105 (HY000): errCode = 2, detailMessage = The statement has been forwarded to master FE() and failed to execute because Master FE is not ready. You may need to check FE's status
```

The error persists until the observer's journal replay catches up (or the observer is restarted), because:

1. The forward target is `Env.getMasterHost()/getMasterRpcPort()` (`MasterOpExecutor` → `FEOpExecutor`), which is backed by the in-memory `masterInfo` field.
2. `masterInfo` is **only** updated by replaying the `OP_MASTER_INFO_CHANGE` journal entry written by the new master (`EditLog.java` → `Env.setMaster()`), or by loading an image at startup. There is no active master re-discovery on the forward path.
3. Meanwhile the old master process is still listening on its ports, so the forwarded thrift call **succeeds at the transport level** — it reaches `StmtExecutor` on the old master, which (as a proxy-forwarded statement, `isProxy=true`) detects it cannot find a valid master and throws the error above (`StmtExecutor.executeByNereids/executeByLegacy`, the `isProxy` branch) instead of the sender learning "this node is not master, retry elsewhere".
4. LB health checks (TCP-level) keep routing client connections to the degraded old master as well, compounding the issue.

Additionally, for `FORWARD_WITH_SYNC` statements (e.g. `CREATE USER`), the old master's error response still carries a `maxJournalId`, so the observer blocks in `JournalObservable.waitOn()` (up to `query_timeout * 1.2`, default ~18 min) before surfacing anything — the client just hangs.

### What You Expected?

After a master failover, the non-master FE should stop forwarding statements to the old (degraded) master quickly — regardless of its journal replay lag. Either:

- the forwarded request should be rejected early by the old master with a NOT_MASTER-style response (so the sender can learn the real master and retry), or
- the sender (`FEOpExecutor`) should re-discover the current master on forward failure/NOT_MASTER and retry against the real master,

so that statements executed via the observer succeed (or fail fast with a clear, actionable error) instead of failing with "Master FE is not ready" (or hanging in journal-sync wait) for the whole `meta_delay_toleration_second` window.

### How to Reproduce?

Reproduced deterministically on Doris 3.0.8 with 4 FEs in docker (fe1=master, fe2/fe3=followers, fe4=observer). `nsenter`-based iptables is used to simulate the degraded states (equivalent to what an OOM/GC-pause produces):

1. **Freeze the observer's journal replay**: block fe4 → fe2/fe3 on the bdbje port (9010), so fe4 cannot replay the new master's journals (its `masterInfo` stays = fe1).
2. **Kill fe1** (`docker kill`, simulating the OOM death of the old master). fe2 is elected as the new master.
3. **Restart fe1 with its bdbje isolated** (block 9010 toward fe2/fe3/fe4). fe1 comes back "alive but degraded": MySQL (9030) and thrift (9020) ports still serve, but it is no longer the master and cannot rejoin/catch up — exactly the state of an OOM-recovered old master that lost the election.
4. **Execute a forward-to-master statement via the observer** (fe4):

```
$ mysql -h -P 9030 -u root -e 'ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "259200")'
ERROR 1105 (HY000) at line 1: errCode = 2, detailMessage = The statement has been forwarded to master FE(172.20.80.2) and failed to execute because Master FE is not ready. You may need to check FE's status
```

(The IP in the message is the **old master's own IP** — the error is thrown by the old master itself.)

Evidence from the logs:

- Observer (fe4) keeps forwarding to the old (dead/degraded) master after the switch:
```
[fe4 log] MasterOpExecutor.forward: forward to master FE TNetworkAddress(hostname:172.20.80.2, port:9020)
```
- Old master (fe1) throws the error on the proxy-forwarded statement:
```
[fe1 log] StmtExecutor.executeByLegacy:1042 execute Exception.
org.apache.doris.common.UserException: errCode = 2, detailMessage = The statement has been forwarded to master FE(172.20.80.2) and failed to execute because Master FE is not ready. You may need to check FE's status
```

Notes:
- With a `FORWARD_WITH_SYNC` statement (`CREATE USER`), the client instead hangs in journal-sync wait on the observer for up to ~18 minutes before failing.
- The recovery observed in production (rolling restart of the observer) matches this analysis: a restart forces an image load + full journal replay, refreshing `masterInfo` to the new master.
- `meta_delay_toleration_second` (default 300s) eventually marks the observer not-ready, which changes the failure mode but does not fix the stale-forward window (5 minutes of broken/hanging statements even in the best case).

### Anything Else?

Related prior work:
- #25371 added `master_address` to thrift responses for BE→FE RPCs so the BE can skip an old master, but the FE→FE `forward()` path never got the equivalent treatment.
- #62721 fixed a result-packet loss during master switch on this same forward path, confirming this area is fragile during master transitions.
- BE-side protection against stale masters exists (e.g. commit f13bd7ad7de "follower fe invalidate dns and be rpc skip old master" probes other FEs after a NOT_MASTER/transport failure), while the FE forward path still relies solely on journal-replay-updated `masterInfo`.

Suggested fix directions (for discussion):
1. In `FrontendServiceImpl.forward()`: when the receiving FE is not the master, return a NOT_MASTER-style response immediately (with the real master address as a hint, similar to #25371) instead of executing the statement deep into `StmtExecutor` and throwing "Master FE is not ready".
2. In `FEOpExecutor.forward()`: on receiving NOT_MASTER (or on forward failure), re-discover the current master (e.g. via `Env.getHaProtocol().getLeader()` which asks bdbje directly and does not depend on journal replay) and retry once against the real master.

This combination would close the stale-forward window regardless of journal replay lag, and also prevents the journal-wait hang caused by the old master's stale `maxJournalId` in error responses.

Full reproduction scripts and timelines available if needed.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with FEOpExecutor.forward(), MasterOpExecutor, FrontendServiceImpl.forward(), and the isProxy branches in StmtExecutor.executeByNereids/executeByLegacy. Reproduce the failover with the described four-FE Docker setup, journal blocking, and ports 9010/9020/9030. Done means forwarding no longer targets the stale master after failover and FORWARD_WITH_SYNC requests fail fast or succeed without the journal-wait hang.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.