Nomad logs format json duplicated with syslog format logs
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Hello guys,
I just want to report a new issue seen by one of my collegue and I, about having nomad logs in json format.
Nomad version: 1.5.3
Consul versio: 1.15.2
OS: Debian 11
With a basic config, log output is fine:
```
{"@level":"info","@message":"blocked evals status modified","@module":"nomad","@timestamp":"2023-04-19T09:45:34.430667Z","paused":false}
{"@level":"info","@message":"initialized keyring","@module":"nomad.keyring","@timestamp":"2023-04-19T09:45:34.436925Z","id":"90209498-e373-124a-6f11-96803efb5202"}
{"@level":"info","@message":"node registration complete","@module":"client","@timestamp":"2023-04-19T09:45:34.605201Z"}
{"@level":"info","@message":"node registration complete","@module":"client","@timestamp":"2023-04-19T09:45:35.609540Z"}
```
Here is the basic config:
```hcl
log_file = "/tmp/nomad.json"
log_rotate_max_files = 5
log_rotate_bytes = 10000000
log_json = true
log_level = "INFO"
```
With an advanced config, log output is duplicated in syslog format AND json format:
```
{"@level":"info","@message":"(runner) starting","@module":"agent","@timestamp":"2023-04-19T09:49:32.571972Z"}
Apr 19 09:49:32 REDACTED nomad[160135]: {"@level":"info","@message":"(runner) starting","@module":"agent","@timestamp":"2023-04-19T09:49:32.571972Z"}
```
Here is the advanced config:
```hcl
# Full configuration options can be found at https://www.nomadproject.io/docs/configuration
data_dir = "REDACTED"
plugin_dir = "REDACTED"
bind_addr = "0.0.0.0"
region = "REDACTED"
datacenter = "REDACTED"
log_level = "INFO"
log_json = true
log_file = "/var/log/nomad/server.log"
log_rotate_max_files = 5
client {
enabled = true
server_join {
retry_join = ["provider=gce project_name=REDACTED tag_value=nomad-server zone_pattern=REDACTED-.*"]
}
node_class = "none"
}
tls {
http = true
rpc = true
ca_file = "REDACTED"
cert_file = "REDACTED"
key_file = "REDACTED"
verify_server_hostname = true
verify_https_client = true
}
vault {
enabled = true
ca_path = "REDACTED"
address = "REDACTED"
}
plugin "docker" {
config {
allow_privileged = true
auth {
config = "/etc/docker/config.json"
}
}
}
acl {
enabled = true
}
consul {
address = "127.0.0.1:8501"
grpc_address = "127.0.0.1:8503"
ssl = true
grpc_ca_file = "REDACTED"
ca_file = "REDACTED"
cert_file = "REDACTED"
key_file = "REDACTED"
token = "REDACTED"
}
telemetry {
publish_allocation_metrics = true
publish_node_metrics = true
prometheus_metrics = true
}
```
We are supposed to have json logs and instead we have a mixed of syslog logs and json logs.
Waiting for your answer.
I will in parallel initiate a ticket with Hashicorp Support Team.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the behavior using the reported Nomad 1.5.3 configurations, especially log_json and log_file, and compare the JSON-only output with the duplicated syslog and JSON output. Trace the logging setup responsible for the two output formats. Done means the configured log file contains JSON logs without unintended syslog-prefixed duplicates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100