HarperFast / HarperFast/harper-pro

Published Dockerfiles set ENV NODE_HOSTNAME=localhost, silently disabling replication after an in-place 4.x→5.x upgrade

Open Beginner friendly
#728 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

All three published Dockerfiles set `ENV NODE_HOSTNAME=localhost`. Because Harper reads environment variables as configuration, that env var becomes `node.hostname` — which `getThisNodeName()` treats as the highest-priority source of node identity. On an in-place 4.x → 5.x upgrade of a clustered node, the node therefore identifies as `localhost`, does not match its row in `system.hdb_nodes`, and **user-database replication silently stops running for that node**.

The operator-facing symptom is worse than the misconfiguration itself: Harper writes the env-derived value back into `harperdb-config.yaml` on every boot, so editing or deleting `node.hostname` in the config file appears to "work" and then reverts on the next restart. There is no obvious signal that an environment variable is the thing overriding it.

## Where

- `Dockerfile:114` — `ENV NODE_HOSTNAME=localhost`
- `Dockerfile-gpu:115` — same
- `Dockerfile-openshift:118` — same

Identity resolution that consumes it (core): `server/nodeName.ts` `getThisNodeName()` checks `node.hostname` first and only falls back to `replication.hostname`, `replication.url`, the TLS CN, and finally the listening port.

## Reproduction

On a two-node 4.x cluster whose nodes are registered in `system.hdb_nodes` as `node-1.example.com` / `node-2.example.com`, with `replication.hostname` correctly set per node:

1. Upgrade one node in place to 5.2.x using the published image.
2. On boot, the log reports:

```
Replication identity mismatch: this node identifies as "localhost" (url wss://localhost:9933)
but no matching row exists in system.hdb_nodes (which has 2 node(s): node-1.example.com,
node-2.example.com). User-database replication will NOT run for this node until its identity
matches a registered node name.
```

3. `grep hostname harperdb-config.yaml` shows both `replication.hostname: node-1.example.com` (correct) and `node.hostname: localhost` (wrong, and it wins).
4. Remove `node.hostname` from `harperdb-config.yaml`, or set it to `node-1.example.com`, and restart → the file is rewritten with `localhost` again. Config file mtime lands ~1s after container start.
5. `docker inspect --format '{{json .Config.Env}}'` shows `NODE_HOSTNAME=localhost`, and `docker history ` attributes it to an `ENV NODE_HOSTNAME=localhost` layer, i.e. it comes from the image rather than the deployment.

Verified against `harperfast/harper-pro:5.2.3`.

## Impact

- Any in-place upgrade of a clustered node to 5.x loses user-database replication for that node until an operator discovers the env var. Replication failing *silently* while the node otherwise serves traffic is the dangerous part — on an active-active pair it presents as data divergence rather than as an upgrade error.
- Every node from these images is affected identically, so an operator fixing node 1 will hit it again on node 2.
- `node.hostname` did not carry this meaning in 4.x, so the env var was inert there. Nothing in an upgrade runbook would flag it.

## Suggested fix

Drop `ENV NODE_HOSTNAME=localhost` from all three Dockerfiles. If a default is wanted for single-node/dev container use, it should not be one that outranks `replication.hostname` on an upgraded cluster member — leaving it unset lets `getThisNodeName()` fall through to `replication.hostname`, which is already correct on a registered node.

Two smaller things that would each have saved most of the debugging time:

1. The mismatch warning is good, and correctly does not tell the operator to cement the wrong value. It could go one step further and name *where* the losing value came from (env var vs config file), since the config file is the misleading place to look.
2. Persisting env-derived config back into `harperdb-config.yaml` makes an env override indistinguishable from a file setting after the first boot. Worth considering whether env-sourced values should be persisted at all.

Contributor guide

Open the contributing guide

Research direction

Start with the ENV NODE_HOSTNAME=localhost lines in Dockerfile, Dockerfile-gpu, and Dockerfile-openshift, then read getThisNodeName() in server/nodeName.ts to confirm the precedence described in the issue. Build or inspect the published image environment and verify that a clustered upgrade can use replication.hostname without the image supplying a localhost override.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, node.js, typescript
Domain
databases, devops, distributed-systems
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.