hashicorp / hashicorp/nomad

nomad operator debug: degrade gracefully on older clusters

Open
#9,568 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When running `nomad operator debug` CLI command against an older cluster,
the command should gracefully handle incompatible API responses.

Not a critical issue, but the expectation is that operators can run `nomad operator debug` using the latest Nomad binary (typically from their workstation), pointed to any Nomad cluster to gather whatever subset of information is available.

Below are example errors encountered when running 1.0.0-rc1 CLI against a 0.9.6 cluster. Almost all are the same underlying issue, documenting for complete checklist to verify after fix.

* agent/host
```client/b547cd3a-085f-68c2-55f4-e99beebb0433: Failed to retrieve agent host data, err: invalid character '<' looking for beginning of value```
* monitor
```monitor: invalid character '<' looking for beginning of value```
* pprof
```...``` HTML page from Nomad UI
* consul-agent-members.json, consul-agent-self.json
```{"Error":"Get \"http:///v1/agent/members\": http: no Host in request URL"}```
* vault-sys-health.json
```{"errors":[]}```
* nomad/0000/csi-volumes.json
```{"Error":"invalid character '\u003c' looking for beginning of value"}```
* nomad/0000/license.json
```{"Error":"invalid character '\u003c' looking for beginning of value"}```
* nomad/0000/plugins.json
```{"Error":"invalid character '\u003c' looking for beginning of value"}```

Correct error response, but the file should not be written:
* nomad/0000/operator-autopilot-health.json
```{"Error":"Unexpected response code: 500 (rpc error: all servers must have raft_protocol set to 3 or higher to use this endpoint)"}```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `nomad operator debug` command and reproduce the listed failures using a latest Nomad binary against an older cluster such as 0.9.6. Use the listed agent, monitor, pprof, Consul, Vault, CSI, license, plugins, and autopilot-health outputs as the verification checklist. Done means incompatible responses are handled gracefully and the invalid autopilot-health response is not written.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.