elastic / elastic/docs-content

[Website]: https://www.elastic.co/docs/release-notes/fleet-server

Open
#7,966 4 comments 0 reactions 0 assignees View on GitHub
documentation source:web Team:Ingest triaged
Dominant language
No language data
Stars
47
Forks
261
Avg merge
3d 12h
Merged PRs (30d)
116

Description

### Before you submit

- [x] This issue is about a documentation page, flow, or piece of content.

### Type of issue

Missing information

### What documentation page or section is affected

https://www.elastic.co/docs/release-notes/fleet-server

### What happened?

## [Docs] Add release notes for the version-specific agent policy check-in churn fix (#280250) in 9.4.5, 9.5.1, and 9.6.0

---
### Summary

[#280250](https://github.com/elastic/kibana/pull/280250) ("[Fleet] Fix agents left with outdated policies by version-specific agent policies", commit [`d1cda8c`](https://github.com/elastic/kibana/commit/d1cda8c39d7ac919240d2d620a3bb63f1aae69e2), closes [#276294](https://github.com/elastic/kibana/issues/276294)) was merged with the **`release_note:skip`** label. As a result it has no entry in any release notes section, in `main` or in either backport branch.

The bug it fixes is user-visible and disruptive: agents assigned to a version-specific policy reported the base policy ID on every check-in, so Fleet Server detected a policy ID mismatch, reassigned the agent, and re-dispatched a `POLICY_CHANGE` action. The agent applied the policy, reported the base ID again, and the cycle repeated indefinitely. Operators see this as Elastic Agent continuously reloading its configuration, and in our case the customer reported it as **"agent constantly restarting"** without any way to correlate it to a known Kibana bug.

The fix shipped to three released/upcoming versions:

| Branch | PR | Target version | Merged |
|---|---|---|---|
| `main` | [#280250](https://github.com/elastic/kibana/pull/280250) | 9.6.0 | 2026-07-29 |
| `9.5` | [#281672](https://github.com/elastic/kibana/pull/281672) | 9.5.1 | 2026-07-30 |
| `9.4` | [#281673](https://github.com/elastic/kibana/pull/281673) | 9.4.5 | 2026-07-30 |

None of these produced a release note entry.

### Why `release_note:skip` looks incorrect here

Every other bug in the same family **is** documented, which makes this omission inconsistent rather than a deliberate editorial choice:

- `docs/release-notes/index.md` 9.4.5 section: `#281195`, `#281092` (version-specific policy fixes)
- 9.5.1 section: `#281195`, `#281092`
- 9.5.0 section: `#274463`, `#281092`, `#281195`

The feature itself was announced in the 9.4.0 release notes (`#258796`), so users who read the notes know the feature exists but get no signal that it shipped with a check-in loop.

This PR is also the largest of the group (711 additions across 6 files) and adds new runtime behaviour that an operator may observe: a periodic orphan sweep in `VersionSpecificPolicyAssignmentTask` that reassigns agents back to the base policy and deletes stale `.fleet-policies` variant documents.

### Requested changes

**1. Add a fix entry under "Data ingestion and Fleet" in the 9.4.5, 9.5.1, and 9.6.0 release notes.**

Both the 9.4.5 and 9.5.1 sections are still in draft (they carry `!!DEFERRED!!` build-candidate comments), so entries can be added before publication. Proposed wording, matching the style of the neighbouring entries:

```markdown
* Fix agents on a version-specific policy reporting the base policy ID on check-in, which made Fleet reassign them and re-send a `POLICY_CHANGE` action on nearly every check-in, causing continuous policy reloads [#280250]({{kib-pull}}280250).
* Fix agents being left on an outdated version-specific policy after the integration that required it is removed from the policy, by reassigning those agents back to the base policy [#280250]({{kib-pull}}280250).
```

If the docs team prefers one bullet per PR, the first bullet alone covers the symptom customers report; the second covers the originally reported issue in #276294.

**2. Add a known issue entry to `docs/release-notes/known-issues.md`.**

Version-specific agent policies were introduced in 9.4.0, so 9.4.0-9.4.4 and 9.5.0 are affected with no fix available on those versions. There is currently no known issue covering this. Proposed entry:

````markdown
::::{dropdown} Agents on version-specific policies reload their configuration continuously

Applies to: {{stack}} 9.4.0-9.4.4, 9.5.0

**Details**

When an agent policy contains an integration or input that declares a minimum {{agent}} version, {{fleet}} creates [version-specific agent policies](docs-content://reference/fleet/version-specific-agent-policies.md) identified as `#`. The version suffix was applied to the agent's assigned `policy_id` but not to the policy ID inside the policy document the agent receives. The agent therefore reported the base policy ID on each check-in, {{fleet-server}} treated this as a policy ID mismatch, reassigned the agent, and re-sent a `POLICY_CHANGE` action. Applying that policy re-created the mismatch, so the cycle repeated indefinitely.

The affected agents keep reloading their configuration and their inputs restart repeatedly, which operators typically observe as {{agent}} restarting continuously. All policies that received a version-specific variant are affected, not only the policy that introduced the version requirement. With {{fleet-server}} debug logging enabled, the loop is visible as repeated `Policy ID mismatch detected, reassigning agent.` messages with `agent_policy_id` set to the base ID and `new_policy_id` set to the `#` variant.

**Workaround**

Upgrade to {{stack}} 9.4.5+ or 9.5.1+. If you cannot upgrade, removing every integration and input that declares a minimum {{agent}} version from the affected agent policies stops the creation of version-specific variants, but on these versions agents already assigned to a variant then remain on an outdated policy (see [#276294](https://github.com/elastic/kibana/issues/276294)).

**Resolved**

This issue is resolved in {{stack}} 9.4.5, 9.5.1, and 9.6.0.
::::
````

**3. Consider a troubleshooting note on the feature page.**

[Version-specific agent policies](https://www.elastic.co/docs/reference/fleet/version-specific-agent-policies) documents the feature but has no troubleshooting or known-issues section. A short pointer from that page to the known issue would help operators who arrive from the symptom rather than from the release notes.

### Notes for the docs team

- The PR description states the new behaviour is gated behind the existing `enableVersionSpecificPolicies` flag. The feature is active on production deployments today (our customer hit this on 9.4.4 with 13 policies affected across their fleet), so please confirm with the Fleet team whether the release note needs a technical-preview qualifier before publishing.
- The `Policy ID mismatch detected, reassigning agent.` log line is emitted by {{fleet-server}} at `debug` level only, which is why the loop is hard to diagnose from default logs. That is worth keeping in the known issue text as the identifying signal.

### Links

- Fix: https://github.com/elastic/kibana/pull/280250
- Commit: https://github.com/elastic/kibana/commit/d1cda8c39d7ac919240d2d620a3bb63f1aae69e2
- Original issue: https://github.com/elastic/kibana/issues/276294
- Backports: https://github.com/elastic/kibana/pull/281672 (9.5), https://github.com/elastic/kibana/pull/281673 (9.4)
- Meta issue: https://github.com/elastic/kibana/issues/281074
- Feature docs: https://www.elastic.co/docs/reference/fleet/version-specific-agent-policies

### Additional info

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the 9.4.5, 9.5.1, and 9.6.0 sections in docs/release-notes/index.md, then review docs/release-notes/known-issues.md and the version-specific agent policies page. Add the requested release-note and known-issue content, and consider the feature-page pointer; done means all affected versions and the documented symptom, workaround, and resolution are covered.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.