hashicorp / hashicorp/consul

Failed check type script warning and missing Stderr logs

Open
#9,089 2 comments 0 reactions 0 assignees View on GitHub
theme/health-checks
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

I've encourtered a couple of issues when implementing `script` type health checks.

* When a command fails, the healthcheck status is in a warning state, rather than failure state.

consul-warning

* The healthcheck output does not contain output written to Stderr, only Stdout. Note, the command is a Python script and is running inside an alpine image in the linked repo, but confirmed that this is also the case for buster.. Healthcheck snippet looks as follows:

```py
# ...

def healthcheck():
conn = http.client.HTTPConnection('127.0.0.1', 8080, timeout=10)
conn.request("GET", "/")
res = conn.getresponse()
assert res.status == 200
print('Stdout')
print("Stderr", file=sys.stderr)
raise Exception("A test exception")

# ...
```

consul-no-stderr

#### Reproduction Steps

Please see source files and instructions in [erichulburd/consul-nomad-issues](https://github.com/erichulburd/consul-nomad-issues).

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 1
checks = 6
services = 6
build:
prerelease =
revision = 1e03567d
version = 1.8.5
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 127.0.0.1:8300
server = true
raft:
applied_index = 135
commit_index = 135
fsm_pending = 0
last_contact = 0
last_log_index = 135
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:9db9a8dc-3426-4424-25e6-eb17b0261b50 Address:127.0.0.1:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 12
goroutines = 91
max_procs = 12
os = darwin
version = go1.14.9
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1

```

Server info

```
agent:
check_monitors = 0
check_ttls = 1
checks = 6
services = 6
build:
prerelease =
revision = 1e03567d
version = 1.8.5
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 127.0.0.1:8300
server = true
raft:
applied_index = 135
commit_index = 135
fsm_pending = 0
last_contact = 0
last_log_index = 135
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:9db9a8dc-3426-4424-25e6-eb17b0261b50 Address:127.0.0.1:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 12
goroutines = 91
max_procs = 12
os = darwin
version = go1.14.9
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1

```

### Operating system and Environment details

MacOS Catalina 10.15.4 is what I'm using to reproduce the errors as documented above (but I'm experiencing similar issues running through OpenStack).

### Log Fragments

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction instructions in erichulburd/consul-nomad-issues and trace Consul's script health-check handling. Inspect how command status and stdout/stderr are collected, then verify that a failing command reports failure and that stderr appears in the health-check output. No specific Consul files or tests are named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.