elastic / elastic/docs-content

Document Kafka dynamic topic with a field default for agent monitoring

Open
#8,056 1 comment 1 reaction 0 assignees View on GitHub
Team:SKI
Dominant language
No language data
Stars
47
Forks
261
Avg merge
3d 12h
Merged PRs (30d)
116

Description

Fleet Kafka users often set a dynamic topic from a custom field they add on integration policies (for example, `service.common_name` via `add_fields`). That works for integration data. Agent monitoring events never get those processors, so the field is missing, topic resolution fails, and monitoring events are dropped (`Dropping event: no topic could be selected`).

The supported pattern is a format string with a default **after** the closing `]`:

```
%{[data_stream.type]}-%{[service.common_name]:agent-monitoring}
```

Integration events go to topics like `logs-`. Monitoring events, which lack the custom field, go to `logs-agent-monitoring` (or `metrics-agent-monitoring`). One Kafka output covers both; you don’t need a separate output per application.

Until recently, Fleet **Dynamic Topic** wrapped free text in `%{[…]}`, so that string could not be entered as a custom value. Wrapping started in 9.1. It is now removed ([PR](https://github.com/elastic/kibana/pull/285581), backported to 9.5 and 9.4), so the string is stored as typed in versions 9.4+ ([backport to 8.19](https://github.com/elastic/kibana/pull/286784/) is pending).

On versions where Dynamic Topic still wraps (9.1–9.3, and 8.19 until the backport), the same format string can be set as a **Static topic**, which does not wrap.

## What to document

### Primary page

[Kafka output settings → Topic settings](https://www.elastic.co/docs/reference/fleet/kafka-output-settings#_topics_settings) (`reference/fleet/kafka-output-settings.md`)

Add:

1. Format strings can include more than one field, and a missing field can have a default: `%{[field]:default}`. The colon must come after `]`. `%{[field:default]}` is invalid.
2. Integration processors do not run on agent monitoring. If the topic depends on a custom field, set a default so monitoring events still get a topic.
3. Example: `%{[data_stream.type]}-%{[service.common_name]:agent-monitoring}`. In the Fleet UI, enter this as a **custom** Dynamic Topic value. Picking a field from the combo box list still wraps that single field in `%{[…]}`; only a custom value is stored as typed.
4. Version scoping:
- In 9.x docs, tag the Dynamic Topic / full format-string behavior with `applies_to` for **9.4+**.
- For **9.1–9.3**, say to put that full string in **Static topic**.
5. Keep the existing `add_fields` example, and make it clear that those fields are not present on monitoring events unless you also use a default (or a separate monitoring output).

### Secondary page

[Kafka output → Topic settings](https://www.elastic.co/docs/reference/fleet/kafka-output#output-kafka-topics-settings) (`reference/fleet/kafka-output.md`). Standalone YAML has no wrapping UI. Add a short note and example of `%{[field]:default}` so the syntax is documented for YAML too. Don’t duplicate the Fleet UI story.

### 8.19 docs

If the [backport PR](https://github.com/elastic/kibana/pull/286784/) to 8.19 is merged, add the same guidance in the [Kafka output settings](https://www.elastic.co/guide/en/fleet/current/kafka-output-settings.html) doc (custom Dynamic Topic stores the format string as typed). Until the backport ships, 8.19 users should use Static topic for the full format string.

## Out of scope

Don’t document this as a product limitation that requires one Kafka output per application. The format-string default is the intended solution.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with reference/fleet/kafka-output-settings.md and read the Topic settings section, then compare reference/fleet/kafka-output.md for the standalone YAML guidance. Document the %{[field]:default} syntax, monitoring-event behavior, Dynamic Topic version differences, and the provided examples without duplicating the Fleet UI details. If the 8.19 backport is merged, apply the corresponding guidance to the current Kafka output settings page.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.