hashicorp / hashicorp/nomad

[feature] - change the filename of logfiles generated for tasks in Nomad

Open
#6,259 1 comment 7 reactions 0 assignees View on GitHub
theme/client theme/logging type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Finding a decent solution for centralized, multi-tenant logging with Nomad is ... difficult ... at the moment, especially when the desire is to keep `nomad logs` working alongside a centralized log shipping solution.

Our current approach is to inject a Filebeat based sidecar per task that adds several additional fields that are absolutely necessary in multi-tenant setups :

```
filebeat.inputs:
- paths:
- /alloc/logs/*.stdout.0
- /alloc/logs/*.stderr.0
encoding: utf-8
fields:
nomad_job: ${NOMAD_JOB_NAME}
nomad_group: ${NOMAD_GROUP_NAME}
nomad_datacenter: ${NOMAD_DC}
nomad_region: ${NOMAD_REGION}
...
```

... but with several hundred sidecars running this is not really feasible any longer :(

**But** if the names of the actual log files generated by Nomad could be changed to include more than just the task name the above problem could be solved by running a **single** log shipper per host.

Inspired by the metrics format that Nomad uses an example could be :

`....stdout.0`

Based on the path and filename we could then route the correct logs to the correct teams and enable much easier correlation between the logs and metrics that Nomad offers.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the `nomad logs` behavior and the task log paths shown in the issue, including `/alloc/logs/*.stdout.0` and `/alloc/logs/*.stderr.0`. Determine where Nomad constructs these filenames and how namespace, job, task group, and task values should be represented; done means the requested naming format works while `nomad logs` continues to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.