influxdata / influxdata/docs-v2
Verify remaining node lifecycle doc claims flagged in PR #7625
- Dominant language
- JavaScript
- Stars
- 82
- Forks
- 326
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 82
Description
## Summary
PR #7625 (Manage the node lifecycle) added the InfluxDB 3 node lifecycle
docs with 14 inline `VERIFY` comments marking claims that were written from
source/design docs rather than confirmed against a live instance, plus two
review findings that need engineering sign-off. This issue tracks the ones
still outstanding after merge so they don't get lost once `PLAN.md` is
deleted.
One `VERIFY (chart)` comment (Helm chart identifiers in the upgrade guide)
has already been resolved and removed in
[323cfe6](https://github.com/influxdata/docs-v2/commit/323cfe6).
## Review findings — reply needed from engineering
From [#7625 (comment)](https://github.com/influxdata/docs-v2/pull/7625#issuecomment-5210031163):
- [ ] **Sequence the compactor and Processing Engine rollouts** —
`content/shared/influxdb3-admin/upgrade.md` (currently around line
368-373, "Release the compactor and the processing engine")
The doc now states explicitly that "Process nodes have no ordering
requirement, so they can roll alongside the compactor" rather than
silently releasing both StatefulSets in the same Helm upgrade. Confirm
with engineering that the Processing Engine genuinely has no ordering
dependency relative to the compactor before treating this as resolved.
- [x] **Set partitions from each StatefulSet replica count** —
`content/shared/influxdb3-admin/upgrade.md`
Appears already addressed: the doc now freezes with
`partition=10000` (a ceiling no real deployment reaches) instead of the
original `partition=99`, which could have let pods with ordinals ≥99
update during the initial image-tag rollout. Worth a final check that
10000 is documented as intentional (it is, via an inline comment) and
that no deployment could plausibly exceed it.
## Remaining `VERIFY (live instance)` comments
All in `content/shared/influxdb3-admin/node-lifecycle.md` unless noted.
Each needs testing against a running Core and/or Enterprise
node/cluster.
- [ ] **Line 54** — Confirm a Core node never surfaces `stopping` or
`removing` (Core lacks the stop/remove commands, but the four-state enum
is shared catalog code). If a Core node can reach `stopping` via the
HTTP endpoint, soften the note.
- [ ] **Line 181** — Confirm the four-step shutdown sequence and its order
for Core: does Core stop accepting writes before flushing rather than
draining in-flight writes first, and does Core mark itself `stopped` in
the catalog on `SIGTERM`? Test: send `SIGTERM` to a Core node, then
query `system.nodes` for its state.
- [ ] **Line 242** — Does reaching `stopped` actually free licensed cores
for reuse? Support cases 00123680 and 00129706 suggest cores may not be
freed by stopping a node. If testing shows cores are only freed on
`remove node` (or not at all), correct both this page and the `stop
node` CLI page.
- [ ] **Line 293** — Two catalog ops (`StopNodeOp` vs. `RequestStopNodeOp`)
return opposite responses (HTTP 400 vs. 200) for a non-running node.
Confirm which path `influxdb3 stop node` uses now, and the response when
the target is `stopping` or `removing` rather than `stopped`.
- [ ] **Line 304** — Scaling down via the stop API has returned an empty
reply (curl exit 52) even when the stop succeeded (suspected race,
EAR #6744). If reproducible, document that an empty reply doesn't mean
the stop failed, and tell automation to confirm via `show nodes`
instead of retrying.
- [ ] **Line 327** — Confirm `stop node --host ` works
for a node whose process is dead (support case 00130867). Also confirm
whether the target node transitions to `stopping` and never reaches
`stopped` (EAR #7030), since no live process completes the handshake.
- [ ] **Line 388** — Confirm the database for
`system.pt_compaction_nodes` (internal notes suggest any querier with
`db=`, not specifically `_internal`). Also confirm the column
names and that this table is not compactor-only, unlike
`system.pt_compaction_active_jobs` and
`system.pt_compaction_deferred_snapshots`.
- [ ] **Line 460** — Confirm the compactor replacement procedure end to
end: stop the compactor, start a new process on different hardware with
the same `--node-id` and `--mode compact`, and confirm it cleanly takes
over the single-writer compaction lease. Also determine whether the
~30s lease TTL forces a wait between stopping the old compactor and
starting the replacement.
- [ ] **Line 494** — **Highest priority**, this claim carries the page:
does a restarting process actually reuse its existing `instance_id`?
Test: `kill -9` a node (leaving it `running` in the catalog), restart
it with the same `--node-id`, confirm it re-registers, and check
whether `instance_id` in `show nodes` changed. If it does *not* reuse
the ID, restarting a node still recorded as `running` would be
refused — invalidating this paragraph, the related troubleshooting
entry, and the "rolling restart is always safe" claim.
- [ ] **Line 745** — Two checks: (1) confirm the catalog sync interval
default of 10 seconds against current config options, and name the
setting if user-tunable; (2) confirm `system.nodes` is reachable via
`--database _internal` with columns `node_id`, `mode`, `state`, and
`updated_at`, and consider whether to also select `instance_id` given
its role in re-registration (same check applies to the Core copy of
this query further down the page).
- [ ] **Line 819** — `+ PRODUCT DECISION`. EAR #7030 (open, seen on
3.11.0) reports `remove node --force-finalize` against a zombie node
causing a self-perpetuating compactor panic loop that kills the
node-removal driver, escalating in one case to an unresponsive
`/health`. If still reproducible, the current warning may be too mild.
Decide with engineering whether to document the stronger caution (and
the `grep 'duplicate catalog subscription'` diagnostic) or wait for a
fix.
## Remaining `VERIFY (eng/product review)` comments
Same underlying question, duplicated in two files that must stay in
sync:
- [ ] **`content/shared/influxdb3-cli/stop/node.md` line 73** and
**`content/influxdb3/enterprise/admin/recover-node.md` line 94** —
Naming of the per-engine drain step. Both were changed from "Parquet:
WAL flush" to "Parquet persistence," on the reading that WAL flush
(the constant `--wal-flush-interval` operation) is distinct from the
Parquet persistence a graceful stop forces. Confirm that reading with
engineering, and confirm "WAL snapshot" is correct for the upgraded
engine. If either file changes, update both.
## Notes
- Line numbers are from the PR #7625 branch as of commit `323cfe6` and
will drift as the page is edited — search for `VERIFY (` to relocate
them.
- `PLAN.md` on the PR branch tracked these items during drafting but is
deleted before merge (repo convention), so this issue is the durable
record.
Contributor guide
No contributing guide indexed for this repository
Research direction
Search for `VERIFY (` in content/shared/influxdb3-admin/node-lifecycle.md, content/shared/influxdb3-admin/upgrade.md, content/shared/influxdb3-cli/stop/node.md, and content/influxdb3/enterprise/admin/recover-node.md. Test the listed Core and Enterprise lifecycle, catalog, compactor, and CLI behaviors against live nodes, beginning with the instance_id restart check and the linked engineering decisions. Done means confirmed claims are documented accurately in both synchronized files, with unsafe or unresolved behavior escalated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- documentation, testing
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100