elastic / elastic/docs-content
Document Kafka dynamic topic with a field default for agent monitoring
- Dominant language
- No language data
- Stars
- 47
- Forks
- 261
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 116
Description
Fleet Kafka users often set a dynamic topic from a custom field they add on integration policies (for example, `service.common_name` via `add_fields`). That works for integration data. Agent monitoring events never get those processors, so the field is missing, topic resolution fails, and monitoring events are dropped (`Dropping event: no topic could be selected`).
The supported pattern is a format string with a default **after** the closing `]`:
```
%{[data_stream.type]}-%{[service.common_name]:agent-monitoring}
```
Integration events go to topics like `logs-`. Monitoring events, which lack the custom field, go to `logs-agent-monitoring` (or `metrics-agent-monitoring`). One Kafka output covers both; you don’t need a separate output per application.
Until recently, Fleet **Dynamic Topic** wrapped free text in `%{[…]}`, so that string could not be entered as a custom value. Wrapping started in 9.1. It is now removed ([PR](https://github.com/elastic/kibana/pull/285581), backported to 9.5 and 9.4), so the string is stored as typed in versions 9.4+ ([backport to 8.19](https://github.com/elastic/kibana/pull/286784/) is pending).
On versions where Dynamic Topic still wraps (9.1–9.3, and 8.19 until the backport), the same format string can be set as a **Static topic**, which does not wrap.
## What to document
### Primary page
[Kafka output settings → Topic settings](https://www.elastic.co/docs/reference/fleet/kafka-output-settings#_topics_settings) (`reference/fleet/kafka-output-settings.md`)
Add:
1. Format strings can include more than one field, and a missing field can have a default: `%{[field]:default}`. The colon must come after `]`. `%{[field:default]}` is invalid.
2. Integration processors do not run on agent monitoring. If the topic depends on a custom field, set a default so monitoring events still get a topic.
3. Example: `%{[data_stream.type]}-%{[service.common_name]:agent-monitoring}`. In the Fleet UI, enter this as a **custom** Dynamic Topic value. Picking a field from the combo box list still wraps that single field in `%{[…]}`; only a custom value is stored as typed.
4. Version scoping:
- In 9.x docs, tag the Dynamic Topic / full format-string behavior with `applies_to` for **9.4+**.
- For **9.1–9.3**, say to put that full string in **Static topic**.
5. Keep the existing `add_fields` example, and make it clear that those fields are not present on monitoring events unless you also use a default (or a separate monitoring output).
### Secondary page
[Kafka output → Topic settings](https://www.elastic.co/docs/reference/fleet/kafka-output#output-kafka-topics-settings) (`reference/fleet/kafka-output.md`). Standalone YAML has no wrapping UI. Add a short note and example of `%{[field]:default}` so the syntax is documented for YAML too. Don’t duplicate the Fleet UI story.
### 8.19 docs
If the [backport PR](https://github.com/elastic/kibana/pull/286784/) to 8.19 is merged, add the same guidance in the [Kafka output settings](https://www.elastic.co/guide/en/fleet/current/kafka-output-settings.html) doc (custom Dynamic Topic stores the format string as typed). Until the backport ships, 8.19 users should use Static topic for the full format string.
## Out of scope
Don’t document this as a product limitation that requires one Kafka output per application. The format-string default is the intended solution.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with reference/fleet/kafka-output-settings.md and read the Topic settings section, then compare reference/fleet/kafka-output.md for the standalone YAML guidance. Document the %{[field]:default} syntax, monitoring-event behavior, Dynamic Topic version differences, and the provided examples without duplicating the Fleet UI details. If the 8.19 backport is merged, apply the corresponding guidance to the current Kafka output settings page.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100