hashicorp / hashicorp/nomad

Nomad logs format json duplicated with syslog format logs

Open
#16,933 1 comment 0 reactions 0 assignees View on GitHub
stage/accepted theme/consul-template theme/logging type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Hello guys,

I just want to report a new issue seen by one of my collegue and I, about having nomad logs in json format.
Nomad version: 1.5.3
Consul versio: 1.15.2
OS: Debian 11

With a basic config, log output is fine:
```
{"@level":"info","@message":"blocked evals status modified","@module":"nomad","@timestamp":"2023-04-19T09:45:34.430667Z","paused":false}
{"@level":"info","@message":"initialized keyring","@module":"nomad.keyring","@timestamp":"2023-04-19T09:45:34.436925Z","id":"90209498-e373-124a-6f11-96803efb5202"}
{"@level":"info","@message":"node registration complete","@module":"client","@timestamp":"2023-04-19T09:45:34.605201Z"}
{"@level":"info","@message":"node registration complete","@module":"client","@timestamp":"2023-04-19T09:45:35.609540Z"}
```

Here is the basic config:
```hcl
log_file = "/tmp/nomad.json"
log_rotate_max_files = 5
log_rotate_bytes = 10000000
log_json = true
log_level = "INFO"
```

With an advanced config, log output is duplicated in syslog format AND json format:
```
{"@level":"info","@message":"(runner) starting","@module":"agent","@timestamp":"2023-04-19T09:49:32.571972Z"}
Apr 19 09:49:32 REDACTED nomad[160135]: {"@level":"info","@message":"(runner) starting","@module":"agent","@timestamp":"2023-04-19T09:49:32.571972Z"}
```

Here is the advanced config:
```hcl
# Full configuration options can be found at https://www.nomadproject.io/docs/configuration

data_dir = "REDACTED"
plugin_dir = "REDACTED"
bind_addr = "0.0.0.0"
region = "REDACTED"
datacenter = "REDACTED"
log_level = "INFO"
log_json = true
log_file = "/var/log/nomad/server.log"
log_rotate_max_files = 5

client {
enabled = true
server_join {
retry_join = ["provider=gce project_name=REDACTED tag_value=nomad-server zone_pattern=REDACTED-.*"]
}
node_class = "none"
}

tls {
http = true
rpc = true

ca_file = "REDACTED"
cert_file = "REDACTED"
key_file = "REDACTED"

verify_server_hostname = true
verify_https_client = true
}

vault {
enabled = true
ca_path = "REDACTED"
address = "REDACTED"
}

plugin "docker" {
config {
allow_privileged = true
auth {
config = "/etc/docker/config.json"
}
}
}

acl {
enabled = true
}

consul {
address = "127.0.0.1:8501"
grpc_address = "127.0.0.1:8503"
ssl = true
grpc_ca_file = "REDACTED"
ca_file = "REDACTED"
cert_file = "REDACTED"
key_file = "REDACTED"
token = "REDACTED"
}

telemetry {
publish_allocation_metrics = true
publish_node_metrics = true
prometheus_metrics = true
}
```

We are supposed to have json logs and instead we have a mixed of syslog logs and json logs.
Waiting for your answer.
I will in parallel initiate a ticket with Hashicorp Support Team.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the behavior using the reported Nomad 1.5.3 configurations, especially log_json and log_file, and compare the JSON-only output with the duplicated syslog and JSON output. Trace the logging setup responsible for the two output formats. Done means the configured log file contains JSON logs without unintended syslog-prefixed duplicates.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.