Graylog2 / Graylog2/graylog-helm

[U-05][J-05] Document upgrades and multi-AZ behavior

Open
#152 3 comments 0 reactions 0 assignees View on GitHub
blocker documentation
Dominant language
Go Template
Stars
12
Forks
3
Avg merge
2d 4h
Merged PRs (30d)
13

Description

## Summary
The chart exposes `updateStrategy.rollingUpdate.partition` and `type: OnDelete`, but the README only lists them in the values tables. There is no procedural upgrade documentation, and nothing covers how availability zones interact with `WaitForFirstConsumer` storage, which pins each existing ordinal to a zone at first volume bind. Add two README sections.

### Details
Current state: `partition` exists for both tiers (`values.yaml:246-250` and 395-399) and renders in both StatefulSets. The README describes it in one line per values table, there is no Upgrading section (Maintenance covers only MongoDB backup and restore), and no AZ content exists in the README or `docs/`. `topologySpreadConstraints` is not exposed. The scheduling knobs are `nodeSelector`, `tolerations`, and `affinity`, and setting `affinity` replaces the default anti-affinity for that tier. One gotcha: `values.schema.json` (lines 352 and 540) types `partition` as string or null, so a plain `--set` with an integer fails validation. Use `--set-string` or a quoted values-file entry.

**Section 1, Upgrading:**

- [ ] Partition canary: set partition to N-1, upgrade, verify the canary via `GET /api/system/cluster/nodes` plus actual throughput, then walk partition down to 0. Same for the Datanode, adding OpenSearch health checks.
- [ ] `type: OnDelete` for full manual cadence control.
- [ ] Note that StatefulSets update from the highest ordinal down, so pod-0 goes last automatically.
- [ ] Mixed-version UI churn during rolls: recommend ingress session affinity via the existing `ingress.web.annotations`, with one controller-specific example.
- [ ] Start a per-release `upgrade-notes.md` with the next release and wire it into `docs/RELEASING.md`.

**Section 2, Multi-AZ:**

- [ ] Zone pinning: PVCs bind where the pod first schedules and zonal volumes cannot cross zones, so existing ordinals need replacement capacity in their bound zone or sit `Pending`. The chart's own AWS gp3 class uses `WaitForFirstConsumer` (`templates/config/sc/aws-gp3.yaml:19`).
- [ ] Non-graceful node loss: roughly 6 minutes before the volume is force-detached, so replacement pods stall at least that long. State the order of magnitude so operators do not read it as a chart bug.
- [ ] A zone-spread example. Decide first: write it with the `affinity` override (documenting that it replaces the default anti-affinity), or add a `topologySpreadConstraints` passthrough to both StatefulSets and document that instead.
- [ ] Scope note: applies to dynamically provisioned per-ordinal PVCs, not `existingClaim` or disabled persistence.

**Reference:** U-05, U-06, J-05 (Production Readiness Review)

### Impact
Operators upgrade without a canary because nothing tells them how, hit the `--set` schema error with no documented workaround, and first learn about zone pinning when a pod sticks in `Pending` after a node failure.

## Notes for maintainers
Both sections go in `charts/graylog/README.md` plus its table of contents. The `partition` schema typing deserves a small follow-up fix to also accept integers. Related: #14 tracks HA behavior changes, while this issue only documents current behavior.

Contributor guide

Open the contributing guide

Research direction

Start with charts/graylog/README.md and its table of contents, then inspect values.yaml, values.schema.json, templates/config/sc/aws-gp3.yaml, and docs/RELEASING.md to verify the documented behavior. Add the requested Upgrading and Multi-AZ sections, examples, and schema workaround, create the per-release upgrade-notes.md entry, and wire it into the release documentation. Done means the README covers both operational procedures and their limitations without changing chart behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, documentation, infrastructure
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.