hashicorp / hashicorp/nomad

No logs by journald if alloc is in pending state

Open
#12,302 4 comments 0 reactions 0 assignees View on GitHub
stage/needs-verification theme/driver/docker theme/logging type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version

`Nomad v1.2.6`

### Operating system and Environment details

```bash
❯ lsb_release -a
LSB Version: n/a
Distributor ID: EndeavourOS
Description: EndeavourOS Linux
Release: rolling
Codename: rolling
```

### Issue

I have configured `journald` as the logging driver for Docker task plugin with the following config:

```hcl
config {
extra_labels = ["job_name", "job_id", "task_group_name", "task_name", "namespace", "node_name", "node_id"]

logging {
type = "journald"
config {
tag = "nomad"
labels = "com.hashicorp.nomad.alloc_id,com.hashicorp.nomad.job_id,com.hashicorp.nomad.job_name,com.hashicorp.nomad.namespace,com.hashicorp.nomad.node_id,com.hashicorp.nomad.node_name,com.hashicorp.nomad.task_group_name,com.hashicorp.nomad.task_name"
}
}
}
```

The issue is, if the `alloc` is in an unhealthy state where the application fails to start because of a runtime error, `nomad alloc logs` or logs on the UI are completely blank. It makes it really hard to debug why the application is failing to start when there are no logs to troubleshoot.

### Reproduction steps

1. Here's a simple `nginx` deployment Job file:

```hcl
job "nginx" {
datacenters = ["dc1"]

group "nginx" {
count = 1

network {
port "http" {
static = 8080
}
}

service {
name = "nginx"
port = "http"
}

task "nginx" {
driver = "docker"

config {
image = "nginx"

ports = ["http"]

volumes = [
"local:/etc/nginx/conf.d",
]
}

template {
data = <

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with the nginx job, the Docker journald logging configuration, and the invalid nginx configuration. Compare the container output in journalctl with nomad alloc logs while the allocation is Pending. Done means startup error logs remain accessible through Nomad’s allocation logs and UI when journald is configured.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
devops, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.