HarperFast / HarperFast/harper-pro
add_node with retain_authorization:true propagates the raw credential object; peer's replication dialer sends 'Authorization: [object Object]' and loops on 1008 — half-connected cluster passes data-only health checks
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
When a node is joined with the documented object form of credentials —
`add_node { …, retain_authorization: true, authorization: { username, password } }` —
the **initiating** node converts the object to a `Basic …` header string before storing
its own `hdb_nodes` record, but the `add_node_back` it sends the target carries the
**raw object** (`replication/setNode.ts`: `targetAddNodeObj.authorization` is assigned
from `req.authorization` *before* the object→`Basic` conversion runs). The target stores
that object verbatim on its `hdb_nodes` record, and its replication dialer later sets
`headers.Authorization = authorization` with no shape check
(`replication/replicationConnection.ts:1838-1841`, 5.2.2 — the parameter is even typed
`authorization?: string` at `:1800`), producing the literal header `[object Object]`.
Over a plain-`ws://` route (no client certificate), an incoming replication connection is
identified only by that header or by an `hdb_nodes` entry keyed by the caller's IP — so
every outbound leg from the target closes with **1008
"no hdb_nodes entry for IP ::ffff:…; run add_node from this node specifying the peer
URL to register it"** and retries roughly every 500 ms, indefinitely.
The result is a **half-connected cluster that looks healthy to data-only probes**: the
direction whose dialer works (the add_node initiator) delivers everything written on it,
so "write on node1, read on node2" succeeds forever, while anything written on node2
never reaches node1 and node2's logs fill with 1008 disconnect warnings. The misleading
reject reason ("run add_node … to register it" — the entry *exists*, it just wasn't
identified) sends the operator in the wrong direction.
## Affected versions
Reproduced on `@harperfast/harper-pro` **5.2.0** and **5.2.2** (npm); the relevant code
is unchanged between them, so 5.2.1 is presumably affected as well. Not tested on 5.1.x.
## Minimal repro (two local nodes, ~2 minutes)
```bash
mkdir -p /tmp/hdb-repro && cd /tmp/hdb-repro
npm init -y >/dev/null && npm i @harperfast/harper-pro@5.2.2 --no-fund --no-audit
HP=/tmp/hdb-repro/node_modules/@harperfast/harper-pro/dist/bin/harper.js
# two isolated nodes: n1 (ops 39125, repl 39127), n2 (ops 39135, repl 39137)
for n in 1 2; do
d=/tmp/hdb-repro/n$n; mkdir -p $d
O=$((39115+n*10)); R=$((39117+n*10)); M=$((39120+n*10))
ROOTPATH=$d HOME=$d HDB_ADMIN_USERNAME=admin HDB_ADMIN_PASSWORD=testpass123 \
THREADS_COUNT=1 OPERATIONSAPI_NETWORK_PORT=$O HTTP_PORT=$((O+1)) \
node $HP install >/dev/null
# per-node identity, insecure local replication port, distinct mqtt ports
node -e "
const Y=require('/tmp/hdb-repro/node_modules/yaml'); // hoisted harper-pro dependency
const fs=require('fs'); const p='$d/harper-config.yaml';
const c=Y.parse(fs.readFileSync(p,'utf8'));
c.replication={hostname:'n$n',url:'ws://127.0.0.1:$R',port:$R,databases:'*'};
c.node={...(c.node||{}),hostname:'n$n'};
c.mqtt.network.port=$M; c.mqtt.network.securePort=$((M+1));
c.logging.level='info';
fs.writeFileSync(p,Y.stringify(c));"
(cd $d && ROOTPATH=$d HOME=$d THREADS_COUNT=1 nohup node $HP run > $d/stdout.log 2>&1 &)
done
until curl -sf -o /dev/null http://127.0.0.1:39125/health && \
curl -sf -o /dev/null http://127.0.0.1:39135/health; do sleep 1; done
# join with the OBJECT credential form
curl -s -u admin:testpass123 -X POST http://127.0.0.1:39125 \
-H 'Content-Type: application/json' -d '{
"operation":"add_node","hostname":"n2","url":"ws://127.0.0.1:39137",
"verify_tls":false,"retain_authorization":true,
"authorization":{"username":"admin","password":"testpass123"}}'
sleep 10
```
Observe:
```bash
# 1) n2 stored the RAW OBJECT; n1 stored the encoded header string
curl -s -u admin:testpass123 -X POST http://127.0.0.1:39135 \
-H 'Content-Type: application/json' -d '{
"operation":"search_by_value","database":"system","table":"hdb_nodes",
"search_attribute":"name","search_value":"*",
"get_attributes":["name","authorization"]}'
# → n1's row on n2: "authorization":{"username":"admin","password":"testpass123"}
# 2) n2's outbound legs loop on 1008 every ~500ms
grep -c 'no hdb_nodes entry for IP' /tmp/hdb-repro/n2/log/hdb.log
# 3) asymmetric cluster: n1→n2 connected, n2→n1 never connects
curl -s -u admin:testpass123 -X POST http://127.0.0.1:39135 \
-H 'Content-Type: application/json' -d '{"operation":"cluster_status"}'
# → database_sockets: [{"database":"system","connected":false}]
# 4) why data-only probes miss it: rows written on n1 DO replicate to n2
# (the intact n2→n1 subscription carries them); rows written on n2 never
# reach n1. A "write here, read there" check on the active node passes.
```
Rerunning the same join with a **preformed header string** —
`"authorization": "Basic YWRtaW46dGVzdHBhc3MxMjM="` — leaves both sides' records in
header form and both directions connect within seconds.
## Observed vs expected
- **Observed:** the two sides of one `add_node` persist different shapes of the same
credential (initiator: `Basic …` string; target: raw object). The dialer sends the
object verbatim as the `Authorization` header; the receiving middleware then rejects
with a reason that tells the operator to run `add_node` — which cannot fix it, since
the record exists and re-running re-propagates the object.
- **Expected:** both sides persist the same, usable credential shape, or the object form
is rejected/coerced at intake. A cluster join that reports success should not leave one
direction permanently unauthorized.
## Workaround
Pass `authorization` as a preformed header string instead of an object:
`"authorization": "Basic "` (one line; both sides then store it
verbatim).
## Suggested fix — make the misuse unrepresentable
In `replication/setNode.ts`, the object→string conversion
(`'Basic ' + base64(username+':'+password)`) currently runs *after*
`targetAddNodeObj` is built. Any of the following closes the hole, in order of
preference:
1. **Normalize at intake:** coerce `req.authorization` to the header string as the first
step of `setNode`, before anything reads it — the operation POST, the local
`hdb_nodes` write, and the propagated `add_node_back` then all carry one shape.
2. **Serialize before propagation:** build `targetAddNodeObj.authorization` from the
already-converted value rather than `req.authorization`, and apply the same
conversion in `addNodeBack` before persisting (defense for older initiators).
3. **Validate at the last use:** `createWebSocket` already types `authorization` as
`string`; enforce it — a non-string should log loudly ("stored node authorization is
not a header string; re-run add_node with a string or credentials object") instead of
sending `[object Object]` and letting the peer's generic 1008 reason misdirect.
Independently, the 1008 reject reason could distinguish "no matching node record" from
"a record exists but the presented Authorization header failed" — the current message
tells the operator to run `add_node` in exactly the case where re-running it re-creates
the problem.
Contributor guide
Research direction
Start in replication/setNode.ts by tracing authorization through the local hdb_nodes write and propagated add_node_back, then inspect replication/replicationConnection.ts:1800 and 1838-1841. Run the supplied two-node reproduction with object credentials and compare it with the preformed-header case. Done means both sides persist a usable header string, both replication directions connect, and the repeated 1008 warnings stop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100