Large number of Nomad logmon subprocesses
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
If filing a bug please include the following:
### Nomad version
Output from `nomad version`
Nomad v0.10.1 (0d4e5d949fe073c47a947ea36bfef31a3c49224f)
### Operating system and Environment details
Debian 9.7
### Issue
We noticed a big spike in CPU on all clients. We've narrowed it down to the 50+ nomad logmon sub processes. Please let me know if there is anything else I can provide.
```
root 36191 79.7 0.1 5879136 137212 ? Ssl 15:46 2:44 /usr/local/bin/nomad agent -config=/etc/nomad/config.hcl
root 36520 0.2 0.0 527832 20924 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36551 0.6 0.0 585172 20404 ? Sl 15:46 0:01 \_ /usr/local/bin/nomad docker_logger
root 36574 0.1 0.0 509776 18280 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36584 0.1 0.0 724184 19724 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36609 0.4 0.0 511184 19376 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36619 0.1 0.0 509520 20012 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36661 0.1 0.0 509520 19296 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36687 0.1 0.0 649300 20456 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36705 0.2 0.0 592216 19036 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36722 0.3 0.0 511440 20120 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36753 0.2 0.0 517972 20848 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36769 0.1 0.0 576720 19768 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36784 0.2 0.0 445904 19988 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36798 0.2 0.0 591704 19548 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 36814 0.6 0.0 510928 19372 ? Sl 15:46 0:01 \_ /usr/local/bin/nomad docker_logger
root 37001 0.1 0.0 517972 19928 ? Sl 15:46 0:00 \_ /usr/local/bin/nomad docker_logger
root 37115 0.4 0.0 511184 19612 ? Sl 15:47 0:00 \_ /usr/local/bin/nomad docker_logger
root 37424 0.3 0.0 584916 19892 ? Sl 15:47 0:00 \_ /usr/local/bin/nomad docker_logger
root 37564 0.1 0.0 444240 20676 ? Sl 15:47 0:00 \_ /usr/local/bin/nomad docker_logger
root 37848 2.0 0.0 650708 19932 ? Sl 15:47 0:03 \_ /usr/local/bin/nomad docker_logger
root 37896 0.3 0.0 584660 18716 ? Sl 15:47 0:00 \_ /usr/local/bin/nomad docker_logger
root 38143 11.3 0.0 658648 18300 ? Sl 15:47 0:20 \_ /usr/local/bin/nomad logmon
root 38148 10.2 0.0 665436 17056 ? Sl 15:47 0:18 \_ /usr/local/bin/nomad logmon
root 38158 11.5 0.0 593112 18324 ? Sl 15:47 0:20 \_ /usr/local/bin/nomad logmon
root 38163 10.3 0.0 658648 19164 ? Sl 15:47 0:18 \_ /usr/local/bin/nomad logmon
root 38184 17.7 0.0 804960 19184 ? Sl 15:47 0:31 \_ /usr/local/bin/nomad logmon
root 38219 10.7 0.0 534364 18296 ? Sl 15:47 0:19 \_ /usr/local/bin/nomad logmon
root 38232 11.2 0.0 583252 18204 ? Sl 15:47 0:19 \_ /usr/local/bin/nomad logmon
root 38244 11.8 0.0 740320 20684 ? Sl 15:47 0:21 \_ /usr/local/bin/nomad logmon
root 38305 12.3 0.0 683492 20228 ? Sl 15:47 0:21 \_ /usr/local/bin/nomad logmon
root 38331 11.6 0.0 806368 19880 ? Sl 15:47 0:20 \_ /usr/local/bin/nomad logmon
root 38340 10.6 0.0 592856 19868 ? Sl 15:47 0:18 \_ /usr/local/bin/nomad logmon
root 38344 11.8 0.0 666588 20316 ? Sl 15:47 0:20 \_ /usr/local/bin/nomad logmon
root 38360 11.2 0.0 656984 19672 ? Sl 15:47 0:19 \_ /usr/local/bin/nomad logmon
root 38366 12.0 0.0 665436 19644 ? Sl 15:47 0:21 \_ /usr/local/bin/nomad logmon
root 38374 10.6 0.0 519124 19736 ? Sl 15:47 0:18 \_ /usr/local/bin/nomad logmon
root 38385 12.3 0.0 658648 19344 ? Sl 15:47 0:21 \_ /usr/local/bin/nomad logmon
root 38427 11.8 0.0 601564 18256 ? Sl 15:47 0:20 \_ /usr/local/bin/nomad logmon
root 38444 11.1 0.0 582996 16988 ? Sl 15:47 0:19 \_ /usr/local/bin/nomad logmon
root 38460 12.5 0.0 593112 19160 ? Sl 15:47 0:22 \_ /usr/local/bin/nomad logmon
root 38465 10.8 0.0 656728 19120 ? Sl 15:47 0:19 \_ /usr/local/bin/nomad logmon
root 38486 14.2 0.0 601308 19488 ? Sl 15:47 0:25 \_ /usr/local/bin/nomad logmon
root 38492 11.5 0.0 591448 18932 ? Sl 15:47 0:20 \_ /usr/local/bin/nomad logmon
root 38515 12.0 0.0 656984 20268 ? Sl 15:47 0:21 \_ /usr/local/bin/nomad logmon
root 38533 10.1 0.0 591704 19068 ? Sl 15:47 0:17 \_ /usr/local/bin/nomad logmon
root 38545 13.6 0.0 593112 19588 ? Sl 15:47 0:23 \_ /usr/local/bin/nomad logmon
root 38559 11.2 0.0 517972 18076 ? Sl 15:47 0:19 \_ /usr/local/bin/nomad logmon
root 38584 10.1 0.0 666844 19224 ? Sl 15:47 0:17 \_ /usr/local/bin/nomad logmon
root 38602 11.6 0.0 879844 18212 ? Sl 15:47 0:20 \_ /usr/local/bin/nomad logmon
root 38603 10.9 0.0 740576 18540 ? Sl 15:47 0:19 \_ /usr/local/bin/nomad logmon
root 38642 13.8 0.0 617700 17168 ? Sl 15:47 0:24 \_ /usr/local/bin/nomad logmon
root 38643 11.8 0.0 666588 20024 ? Sl 15:47 0:20 \_ /usr/local/bin/nomad logmon
```
### Reproduction steps
This only appears to be happening in our dev cluster (v0.10.1). Our production cluster is running (v0.9.6) and is not experiencing this issue.
### Job file (if appropriate)
### Nomad Client logs (if appropriate)
If possible please post relevant logs in the issue.
Logs and other artifacts may also be sent to: nomad-oss-debug@hashicorp.com
Please link to your Github issue in the email and reference it in the subject
line:
> To: nomad-oss-debug@hashicorp.com
>
> Subject: GH-1234: Errors garbage collecting allocs
### Nomad Server logs (if appropriate)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by examining the Nomad v0.10.1 client configuration at /etc/nomad/config.hcl and the reported logmon and docker_logger subprocess behavior. Reproduce the CPU spike in the Debian 9.7 development cluster, compare it with the v0.9.6 production cluster, and use client logs to identify why so many logmon processes are spawned. Done means the cause is confirmed and excessive subprocess CPU is no longer observed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100