hashicorp / hashicorp/nomad

Nomad servers memory leak and all allocations restarts after "nomad system gc"

Open
#18,063 5 comments 0 reactions 0 assignees View on GitHub
stage/needs-investigation theme/core type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.6.0
BuildDate 2023-07-18T18:51:11Z
Revision 87d411f4e3a6a744d328c95708c60c12bae0fb18

### Operating system and Environment details
Fedora release 35 (Thirty Five)

### Issue
We have 3 master nodes cluster with 2 CPU, 4 RAM each. We have 30 client nodes.
The resourses was enough before, but we started to see memory leak - during a week the RAM is exausted on each of the node.
We started to see it on 1.5.6 version, we did not face this before this 1.5.6.
Recently we updated 1.5.6 to 1.6.0 hoping that this problem will dissapear.
You can see on 07/20 is our update time for new 1.6.0 version:
![image](https://github.com/hashicorp/nomad/assets/20965960/7773bb3e-0bed-4b10-96a0-4117d7cdc9fe)
But a week passed, and all 3 servers were almost out of RAM.
We didn't find anything related in logs, except some raft error messages (log files in the attachments), so we decided to run "nomad system gc" on server-1 hoping that some resources will be freed.
But after that, we lost connection to the server-1 and some client nodes started to fail heartbeats and allocations on them were restarted. We did not see this behaviour before 1.5.6 version. It's unacceptable.

### Reproduction steps
"nomad system gc" in exausted resoures situation.

#### Expected Result
Free some RAM, cluster not restarts allocations.

#### Actual Result
During freeing RAM, cluster fails client node heartbeats and restarts allocations.
We think all this is related to:
https://github.com/hashicorp/nomad/issues/17973
https://github.com/hashicorp/nomad/issues/17974
Because the node server-1 on which we run "nomad system gc" was out of resources and stopped to respond to the cluster.

But we can't find the reason of memory leak.
Please advise what we can do to avoid it.

### Nomad Server logs for last 24h
"nomad system gc" was run at was run at 2023-07-26T04:34:00 on server-1. After that we lost connection to the server for a while.
[server1.log](https://github.com/hashicorp/nomad/files/12168183/server1.log)
[server2-leader.log](https://github.com/hashicorp/nomad/files/12168184/server2-leader.log)
[server3.log](https://github.com/hashicorp/nomad/files/12168186/server3.log)

### Nomad Client logs for last 24h
Multiple clients disconnected, here's log of one of them:
[client1.log](https://github.com/hashicorp/nomad/files/12168193/client1.log)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reproduction entry point, `nomad system gc` under exhausted resources, and review the attached server and client logs from 2023-07-26. Compare the behavior with linked issues #17973 and #17974. Done means identifying the memory leak and preventing garbage collection from causing server heartbeat failures or allocation restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.