Failed check type script warning and missing Stderr logs
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
I've encourtered a couple of issues when implementing `script` type health checks.
* When a command fails, the healthcheck status is in a warning state, rather than failure state.

* The healthcheck output does not contain output written to Stderr, only Stdout. Note, the command is a Python script and is running inside an alpine image in the linked repo, but confirmed that this is also the case for buster.. Healthcheck snippet looks as follows:
```py
# ...
def healthcheck():
conn = http.client.HTTPConnection('127.0.0.1', 8080, timeout=10)
conn.request("GET", "/")
res = conn.getresponse()
assert res.status == 200
print('Stdout')
print("Stderr", file=sys.stderr)
raise Exception("A test exception")
# ...
```

#### Reproduction Steps
Please see source files and instructions in [erichulburd/consul-nomad-issues](https://github.com/erichulburd/consul-nomad-issues).
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 1
checks = 6
services = 6
build:
prerelease =
revision = 1e03567d
version = 1.8.5
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 127.0.0.1:8300
server = true
raft:
applied_index = 135
commit_index = 135
fsm_pending = 0
last_contact = 0
last_log_index = 135
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:9db9a8dc-3426-4424-25e6-eb17b0261b50 Address:127.0.0.1:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 12
goroutines = 91
max_procs = 12
os = darwin
version = go1.14.9
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```
Server info
```
agent:
check_monitors = 0
check_ttls = 1
checks = 6
services = 6
build:
prerelease =
revision = 1e03567d
version = 1.8.5
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 127.0.0.1:8300
server = true
raft:
applied_index = 135
commit_index = 135
fsm_pending = 0
last_contact = 0
last_log_index = 135
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:9db9a8dc-3426-4424-25e6-eb17b0261b50 Address:127.0.0.1:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 12
goroutines = 91
max_procs = 12
os = darwin
version = go1.14.9
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```
### Operating system and Environment details
MacOS Catalina 10.15.4 is what I'm using to reproduce the errors as documented above (but I'm experiencing similar issues running through OpenStack).
### Log Fragments
Contributor guide
Research direction
Start with the reproduction instructions in erichulburd/consul-nomad-issues and trace Consul's script health-check handling. Inspect how command status and stdout/stderr are collected, then verify that a failing command reports failure and that stderr appears in the health-check output. No specific Consul files or tests are named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100