elastic / elastic/fleet-server

Decouple checkin_limit.max from memory-based tier selection

Open
#7,713 0 comments 0 reactions 0 assignees View on GitHub
enhancement Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
113
Forks
117
Avg merge
1d 16h
Merged PRs (30d)
112

Description

## Describe the enhancement

`checkin_limit.max` (the concurrent long-poll connection cap that, when exceeded, returns HTTP 429 `MaxLimit`) is currently coupled to cache sizing through the shared tier system in `internal/pkg/config/`. Both are selected by the same `loadLimits()` call:

```go
// internal/pkg/config/config.go
agentLimits := loadLimits(log, fleetInput.Server.Limits.MaxAgents)
fleetInput.Cache.LoadLimits(agentLimits) // cache: num_counters, max_cost
fleetInput.Server.Limits.LoadLimits(agentLimits) // server: checkin_limit.max, etc.
```

When `max_agents` is unset (the default), `loadLimits` falls through to `memEnvLimits`, which selects the tier by available container RAM. PRs #7568 and #7573 correctly changed `memMB()` to read container RAM (via `GOMEMLIMIT` / cgroup) rather than host RAM — this was the right fix for OOMKills caused by oversized ristretto caches. However, because the same tier drives both cache sizing _and_ `checkin_limit.max`, Fleet Server instances running in small containers on large hosts now select a lower checkin limit tier than they did before those PRs.

Concrete example: a 4 GB container previously read the host's 32+ GB RAM and selected the `max` tier (`checkin_limit.max = 80,000`). It now correctly reads 4 GB and selects `lte10000` (`checkin_limit.max = 10,000`). The cache sizing change is correct. The checkin limit change is an unintended regression for deployments that spread agents across multiple small-container nodes.

## The coupling is not necessary

Cache sizing and checkin capacity have different natural inputs:

- **Cache** (`max_cost`, `num_counters`): should be bounded by available container memory — reading container RAM is correct.
- **`checkin_limit.max`**: is a concurrent-connection semaphore (not a pre-allocation). Its natural input is expected agent count, not RAM. No memory is reserved for unused semaphore slots.

Memory analysis confirms bumping `checkin_limit.max` is safe:
- `semaphore.NewWeighted(N)` allocates a fixed-size struct regardless of `N`; slots are never pre-allocated
- Actual per-agent RAM (goroutine stacks ~2 KB, `checkin.Bulk.pending` entries ~250 B, policy/action subscription nodes ~450 B) scales with _actual_ concurrent connections, not the tier ceiling
- The ristretto `max_cost` tier values for `lte0500` through `lte10000` are identical (all 50 MiB), meaning cache sizing already doesn't benefit from the coupled tier selection in the lower half of the range

## Proposed solution

Decouple the RAM-based path from the checkin limit selection. Two options worth evaluating:

1. **Split the tier lookup**: use `containerMemoryMB()` only for cache-related fields and a separate agent-count signal for `checkin_limit.max`. When `max_agents` is unset, default `checkin_limit.max` to a high/unlimited value (as `defaultCheckinMax = 0` already does) and let `max_agents` remain the explicit knob.

2. **Keep a single tier but source it from agent count, not RAM**: always use `max_agents` (requiring operators to set it explicitly) and remove the RAM-based auto-selection of the server limits. Cache sizing would use a separate, RAM-only lookup.

Either way, `max_agents` should be clearly documented as the primary knob for `checkin_limit.max`, with the RAM-based path as a conservative fallback for cache sizing only.

## Related

- #7568 — introduced `containerMemoryMB()` (cache OOMKill fix)
- #7573 — added cgroup fallback to `containerMemoryMB()`

Contributor guide

Open the contributing guide

Research direction

Start in internal/pkg/config/config.go by tracing loadLimits, memEnvLimits, and containerMemoryMB, then follow how FleetInput.Cache.LoadLimits and Server.Limits.LoadLimits consume the shared tier. Compare the two proposed decoupling approaches and verify the chosen behavior against the checkin limit and cache-sizing paths; done means RAM still controls cache sizing while max_agents clearly controls checkin_limit.max.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.