Add allocation resource utilization to the /client/stats endpoint
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Problem
Currently the `/client/stats` endpoint has a variety of utilization information including `CPUTicksConsumed` and `Memory.Used` which can be used to determine percent utilization for the host. This endpoint cannot currently be used to determine the percent utilization for a host _limited to allocations_.
The output for `nomad node status` contains the following, which includes allocated resource utilization:
```shellsession
Allocated Resources
CPU Memory Disk
1000/38400 MHz 512 MiB/32 GiB 300 MiB/292 GiB
Allocation Resource Utilization
CPU Memory
14/38400 MHz 1.9 MiB/32 GiB
Host Resource Utilization
CPU Memory Disk
2258/38400 MHz 22 GiB/32 GiB 156 GiB/466 GiB
Allocations
ID Node ID Task Group Version Desired Status Created Modified
3029bd2a e939dc3a cache 0 run running 6s ago 4s ago
```
It achieves this by first getting the allocations on the client and then aggregating their individual resource utilization via `/client/allocation/:id/stats`
### Proposal
Move this aggregating logic into the API layer.
This would allow the UI to also present this information without making an excessive amount of API requests (especially considering the UI polls this endpoint on a 2s interval).
**Consideration: ACLs**
Allocation stats are dictated by the `namespace:read-job` permission while client stats are dictated by `node:read`. As part of this proposal, we're acknowledging that allocation stats _in aggregate_ are acceptable to read with the `node:read` permission.
**Response Shape**
The allocation stats response already aggregates the stats figures and returns the shape:
```json
{
"ResourceUsage": {
"CpuStats": {
"Measured": ["Throttled Periods", "Throttled Time", "Percent"],
"Percent": 0.14159538847117795,
"SystemMode": 0,
"ThrottledPeriods": 0,
"ThrottledTime": 0,
"TotalTicks": 3.256693934837093,
"UserMode": 0
},
"MemoryStats": {
"Cache": 1744896,
"KernelMaxUsage": 0,
"KernelUsage": 0,
"MaxUsage": 4710400,
"Measured": ["RSS", "Cache", "Swap", "Max Usage"],
"RSS": 1486848,
"Swap": 0
}
}
}
```
The client stats response can take this same shape and further aggregate all allocations. The property name should be something like `AllocatedResourceUsage` or `AllocationResourceUsage`
**Hidden benefit**
As @cgbaker pointed out, aggregating all alloc stats at once on the client saves us from round-tripping from the server to the client N times as is currently the case with the CLI implementation.
### Related Issues
#6892
#8694
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the `/client/stats` endpoint and the existing `/client/allocation/:id/stats` aggregation used by `nomad node status`. Confirm how client and allocation ACLs are enforced, then define completion as one client-stats response aggregating all allocations with the documented `ResourceUsage` shape and no per-allocation round trips.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100