HarperFast / HarperFast/harper-pro

add_node with retain_authorization:true propagates the raw credential object; peer's replication dialer sends 'Authorization: [object Object]' and loops on 1008 — half-connected cluster passes data-only health checks

Open
#849 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

When a node is joined with the documented object form of credentials —
`add_node { …, retain_authorization: true, authorization: { username, password } }` —
the **initiating** node converts the object to a `Basic …` header string before storing
its own `hdb_nodes` record, but the `add_node_back` it sends the target carries the
**raw object** (`replication/setNode.ts`: `targetAddNodeObj.authorization` is assigned
from `req.authorization` *before* the object→`Basic` conversion runs). The target stores
that object verbatim on its `hdb_nodes` record, and its replication dialer later sets
`headers.Authorization = authorization` with no shape check
(`replication/replicationConnection.ts:1838-1841`, 5.2.2 — the parameter is even typed
`authorization?: string` at `:1800`), producing the literal header `[object Object]`.

Over a plain-`ws://` route (no client certificate), an incoming replication connection is
identified only by that header or by an `hdb_nodes` entry keyed by the caller's IP — so
every outbound leg from the target closes with **1008
"no hdb_nodes entry for IP ::ffff:…; run add_node from this node specifying the peer
URL to register it"** and retries roughly every 500 ms, indefinitely.

The result is a **half-connected cluster that looks healthy to data-only probes**: the
direction whose dialer works (the add_node initiator) delivers everything written on it,
so "write on node1, read on node2" succeeds forever, while anything written on node2
never reaches node1 and node2's logs fill with 1008 disconnect warnings. The misleading
reject reason ("run add_node … to register it" — the entry *exists*, it just wasn't
identified) sends the operator in the wrong direction.

## Affected versions

Reproduced on `@harperfast/harper-pro` **5.2.0** and **5.2.2** (npm); the relevant code
is unchanged between them, so 5.2.1 is presumably affected as well. Not tested on 5.1.x.

## Minimal repro (two local nodes, ~2 minutes)

```bash
mkdir -p /tmp/hdb-repro && cd /tmp/hdb-repro
npm init -y >/dev/null && npm i @harperfast/harper-pro@5.2.2 --no-fund --no-audit
HP=/tmp/hdb-repro/node_modules/@harperfast/harper-pro/dist/bin/harper.js

# two isolated nodes: n1 (ops 39125, repl 39127), n2 (ops 39135, repl 39137)
for n in 1 2; do
d=/tmp/hdb-repro/n$n; mkdir -p $d
O=$((39115+n*10)); R=$((39117+n*10)); M=$((39120+n*10))
ROOTPATH=$d HOME=$d HDB_ADMIN_USERNAME=admin HDB_ADMIN_PASSWORD=testpass123 \
THREADS_COUNT=1 OPERATIONSAPI_NETWORK_PORT=$O HTTP_PORT=$((O+1)) \
node $HP install >/dev/null
# per-node identity, insecure local replication port, distinct mqtt ports
node -e "
const Y=require('/tmp/hdb-repro/node_modules/yaml'); // hoisted harper-pro dependency
const fs=require('fs'); const p='$d/harper-config.yaml';
const c=Y.parse(fs.readFileSync(p,'utf8'));
c.replication={hostname:'n$n',url:'ws://127.0.0.1:$R',port:$R,databases:'*'};
c.node={...(c.node||{}),hostname:'n$n'};
c.mqtt.network.port=$M; c.mqtt.network.securePort=$((M+1));
c.logging.level='info';
fs.writeFileSync(p,Y.stringify(c));"
(cd $d && ROOTPATH=$d HOME=$d THREADS_COUNT=1 nohup node $HP run > $d/stdout.log 2>&1 &)
done
until curl -sf -o /dev/null http://127.0.0.1:39125/health && \
curl -sf -o /dev/null http://127.0.0.1:39135/health; do sleep 1; done

# join with the OBJECT credential form
curl -s -u admin:testpass123 -X POST http://127.0.0.1:39125 \
-H 'Content-Type: application/json' -d '{
"operation":"add_node","hostname":"n2","url":"ws://127.0.0.1:39137",
"verify_tls":false,"retain_authorization":true,
"authorization":{"username":"admin","password":"testpass123"}}'
sleep 10
```

Observe:

```bash
# 1) n2 stored the RAW OBJECT; n1 stored the encoded header string
curl -s -u admin:testpass123 -X POST http://127.0.0.1:39135 \
-H 'Content-Type: application/json' -d '{
"operation":"search_by_value","database":"system","table":"hdb_nodes",
"search_attribute":"name","search_value":"*",
"get_attributes":["name","authorization"]}'
# → n1's row on n2: "authorization":{"username":"admin","password":"testpass123"}

# 2) n2's outbound legs loop on 1008 every ~500ms
grep -c 'no hdb_nodes entry for IP' /tmp/hdb-repro/n2/log/hdb.log

# 3) asymmetric cluster: n1→n2 connected, n2→n1 never connects
curl -s -u admin:testpass123 -X POST http://127.0.0.1:39135 \
-H 'Content-Type: application/json' -d '{"operation":"cluster_status"}'
# → database_sockets: [{"database":"system","connected":false}]

# 4) why data-only probes miss it: rows written on n1 DO replicate to n2
# (the intact n2→n1 subscription carries them); rows written on n2 never
# reach n1. A "write here, read there" check on the active node passes.
```

Rerunning the same join with a **preformed header string** —
`"authorization": "Basic YWRtaW46dGVzdHBhc3MxMjM="` — leaves both sides' records in
header form and both directions connect within seconds.

## Observed vs expected

- **Observed:** the two sides of one `add_node` persist different shapes of the same
credential (initiator: `Basic …` string; target: raw object). The dialer sends the
object verbatim as the `Authorization` header; the receiving middleware then rejects
with a reason that tells the operator to run `add_node` — which cannot fix it, since
the record exists and re-running re-propagates the object.
- **Expected:** both sides persist the same, usable credential shape, or the object form
is rejected/coerced at intake. A cluster join that reports success should not leave one
direction permanently unauthorized.

## Workaround

Pass `authorization` as a preformed header string instead of an object:
`"authorization": "Basic "` (one line; both sides then store it
verbatim).

## Suggested fix — make the misuse unrepresentable

In `replication/setNode.ts`, the object→string conversion
(`'Basic ' + base64(username+':'+password)`) currently runs *after*
`targetAddNodeObj` is built. Any of the following closes the hole, in order of
preference:

1. **Normalize at intake:** coerce `req.authorization` to the header string as the first
step of `setNode`, before anything reads it — the operation POST, the local
`hdb_nodes` write, and the propagated `add_node_back` then all carry one shape.
2. **Serialize before propagation:** build `targetAddNodeObj.authorization` from the
already-converted value rather than `req.authorization`, and apply the same
conversion in `addNodeBack` before persisting (defense for older initiators).
3. **Validate at the last use:** `createWebSocket` already types `authorization` as
`string`; enforce it — a non-string should log loudly ("stored node authorization is
not a header string; re-run add_node with a string or credentials object") instead of
sending `[object Object]` and letting the peer's generic 1008 reason misdirect.

Independently, the 1008 reject reason could distinguish "no matching node record" from
"a record exists but the presented Authorization header failed" — the current message
tells the operator to run `add_node` in exactly the case where re-running it re-creates
the problem.

Contributor guide

Open the contributing guide

Research direction

Start in replication/setNode.ts by tracing authorization through the local hdb_nodes write and propagated add_node_back, then inspect replication/replicationConnection.ts:1800 and 1838-1841. Run the supplied two-node reproduction with object credentials and compare it with the preformed-header case. Done means both sides persist a usable header string, both replication directions connect, and the repeated 1008 warnings stop.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.