apache / apache/doris

[Bug] FE Observer node fails to join cluster in cloud mode with Docker host network

Open
#61,536 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
522

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

Doris 4.0.3, 4.0.4-slim

### What's Wrong?

In Doris cloud mode, when deploying FE cluster using Docker host network where each FE node uses a
different http_port, Observer nodes fail to join the cluster.

Error log:
WARN [Env.getFeNodeTypeAndNameFromHelpers():1520] failed to get fe node type from helper node:
HostInfo{host='127.0.0.1', port=9010}.
java.net.ConnectException: Connection refused
WARN [Env.getClusterIdAndRole():1342] current node HostInfo{host='127.0.0.1', port=9011} is not added to
the group. please add it first.

Root cause:

In Env.getFeNodeTypeAndNameFromHelpers() method:
String url = "http://" + NetUtils.getHostPortInAccessibleFormat(
helperNode.getHost(), // "127.0.0.1" (correct)
Config.http_port // Uses current node's http_port, NOT helper's!
) + "/role?host=...";

The code uses Config.http_port (current node's port) instead of the helper node's http_port. This assumes
all FE nodes use the same http_port, which fails when:
- Using Docker host network mode (containers share host network, must use different ports)

---
What You Expected

FE-2 (Observer) should connect to FE-1's HTTP endpoint at port 8030 and join the cluster successfully.

The code should use the helper node's http_port when constructing the HTTP URL, not the current node's
Config.http_port.

### What You Expected?

1. FE-2 should successfully connect to FE-1's HTTP endpoint and join the cluster as an Observer node.
2. The code should use the helper node's http_port when constructing the HTTP URL, not the current node's
Config.http_port.

Current behavior (wrong):
FE-2 tries: http://127.0.0.1:8031/role (FE-2's own http_port, no service)
Should be: http://127.0.0.1:8030/role (FE-1's http_port, correct)

Suggested fix:
- Store http_port in Meta Service when registering FE nodes
- Or provide a way to specify helper node's http_port in the --helper parameter (e.g., --helper
host:http_port:edit_log_port)

### How to Reproduce?

1. Deploy Doris cloud mode with Meta Service + FoundationDB
2. Configure FE-1 (Master) with http_port=8030, edit_log_port=9010
3. Configure FE-2 (Observer) with http_port=8031, edit_log_port=9011
4. Start FE-1 successfully
5. Start FE-2 with --helper 127.0.0.1:9010
6. FE-2 fails to join with "Connection refused" error

Configuration example:

┌──────┬──────────┬───────────────┬───────────┐
│ Node │ Role │ edit_log_port │ http_port │
├──────┼──────────┼───────────────┼───────────┤
│ FE-1 │ Master │ 9010 │ 8030 │
├──────┼──────────┼───────────────┼───────────┤
│ FE-2 │ Observer │ 9011 │ 8031 │
└──────┴──────────┴───────────────┴───────────┘

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with Env.getFeNodeTypeAndNameFromHelpers() and reproduce the Docker host-network setup using the FE-1 and FE-2 port configuration in the issue. Trace how helper node information is registered and how the HTTP URL is built. Done means the Observer connects to FE-1's HTTP endpoint and joins the cluster successfully with different http_port values.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java
Domain
cloud, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.