hashicorp / hashicorp/nomad

Add allocation resource utilization to the /client/stats endpoint

Open
#9,899 1 comment 2 reactions 0 assignees View on GitHub
stage/accepted theme/api
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Problem

Currently the `/client/stats` endpoint has a variety of utilization information including `CPUTicksConsumed` and `Memory.Used` which can be used to determine percent utilization for the host. This endpoint cannot currently be used to determine the percent utilization for a host _limited to allocations_.

The output for `nomad node status` contains the following, which includes allocated resource utilization:

```shellsession
Allocated Resources
CPU Memory Disk
1000/38400 MHz 512 MiB/32 GiB 300 MiB/292 GiB

Allocation Resource Utilization
CPU Memory
14/38400 MHz 1.9 MiB/32 GiB

Host Resource Utilization
CPU Memory Disk
2258/38400 MHz 22 GiB/32 GiB 156 GiB/466 GiB

Allocations
ID Node ID Task Group Version Desired Status Created Modified
3029bd2a e939dc3a cache 0 run running 6s ago 4s ago
```

It achieves this by first getting the allocations on the client and then aggregating their individual resource utilization via `/client/allocation/:id/stats`

### Proposal

Move this aggregating logic into the API layer.

This would allow the UI to also present this information without making an excessive amount of API requests (especially considering the UI polls this endpoint on a 2s interval).

**Consideration: ACLs**
Allocation stats are dictated by the `namespace:read-job` permission while client stats are dictated by `node:read`. As part of this proposal, we're acknowledging that allocation stats _in aggregate_ are acceptable to read with the `node:read` permission.

**Response Shape**
The allocation stats response already aggregates the stats figures and returns the shape:

```json
{
"ResourceUsage": {
"CpuStats": {
"Measured": ["Throttled Periods", "Throttled Time", "Percent"],
"Percent": 0.14159538847117795,
"SystemMode": 0,
"ThrottledPeriods": 0,
"ThrottledTime": 0,
"TotalTicks": 3.256693934837093,
"UserMode": 0
},
"MemoryStats": {
"Cache": 1744896,
"KernelMaxUsage": 0,
"KernelUsage": 0,
"MaxUsage": 4710400,
"Measured": ["RSS", "Cache", "Swap", "Max Usage"],
"RSS": 1486848,
"Swap": 0
}
}
}
```

The client stats response can take this same shape and further aggregate all allocations. The property name should be something like `AllocatedResourceUsage` or `AllocationResourceUsage`

**Hidden benefit**
As @cgbaker pointed out, aggregating all alloc stats at once on the client saves us from round-tripping from the server to the client N times as is currently the case with the CLI implementation.

### Related Issues

#6892
#8694

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the `/client/stats` endpoint and the existing `/client/allocation/:id/stats` aggregation used by `nomad node status`. Confirm how client and allocation ACLs are enforced, then define completion as one client-stats response aggregating all allocations with the documented `ResourceUsage` shape and no per-allocation round trips.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.