elastic / elastic/integrations
[osquery_manager] Expose ecs_mapping configuration in stream template for scheduled queries
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
Osquery Manager [osquery_manager]
### Dataset Name
osquery_manager.result
### Integration Version
1.23.0
### Agent Version
8.x / 9.x (all versions with osquerybeat)
### OS Version and Architecture
All platforms
### User Goal
I need to inject additional metadata fields into osquery scheduled query results. For example, adding labels, tags, or other context that helps correlate osquery results with organizational metadata (team ownership, environment, compliance scope, etc.).
Osquerybeat already fully supports an `ecs_mapping` configuration option on each stream that can:
- **Remap** osquery result fields to arbitrary destination fields using `field`
- **Inject static/constant values** into the event using `value`
This is defined in the [`StreamConfig` struct](https://github.com/elastic/beats/blob/eff088939b9a4c2f1616a9b64eb5fe4a3352fea4/x-pack/osquerybeat/internal/config/config.go#L32-L40) and processed by [`flattenECSMapping()`](https://github.com/elastic/beats/blob/eff088939b9a4c2f1616a9b64eb5fe4a3352fea4/x-pack/osquerybeat/beater/config_plugin.go#L287-L352) in `config_plugin.go`. The mapping is applied in [`ecs/mapping.go` via the `Map()` method](https://github.com/elastic/beats/blob/eff088939b9a4c2f1616a9b64eb5fe4a3352fea4/x-pack/osquerybeat/internal/ecs/mapping.go#L23-L37), which creates new fields in the event document. Despite the name, it does not restrict destination fields to ECS — any arbitrary dotted key path works.
However, the osquery_manager integration's stream template (`stream.yml.hbs`) only exposes the `query` variable:
```yaml
query: {{query}}
```
There is no way to pass `ecs_mapping` through the Fleet policy to osquerybeat for scheduled queries.
### Existing Features
The current feature set does not satisfy this need because:
1. **No processor support:** Unlike other Fleet-managed beats, the osquery_manager integration does not expose an `add_fields` or other processor configuration in its stream template. Most other integrations allow processors to be configured at the stream level to enrich events.
2. **`ecs_mapping` exists but is unreachable:** Osquerybeat already implements the feature — scheduled queries can use `ecs_mapping` with static `value` entries to inject arbitrary fields. But the integration template never passes this configuration through. See [`StreamConfig.ECSMapping`](https://github.com/elastic/beats/blob/eff088939b9a4c2f1616a9b64eb5fe4a3352fea4/x-pack/osquerybeat/internal/config/config.go#L38-L39) and its use in [`config_plugin.Set()`](https://github.com/elastic/beats/blob/eff088939b9a4c2f1616a9b64eb5fe4a3352fea4/x-pack/osquerybeat/beater/config_plugin.go#L167-L281).
3. **Live queries work, scheduled don't:** Ad-hoc live queries sent via the Kibana Osquery UI *do* support `ecs_mapping` (passed in the [action request payload](https://github.com/elastic/beats/blob/eff088939b9a4c2f1616a9b64eb5fe4a3352fea4/x-pack/osquerybeat/internal/action/action.go#L45-L66)). Scheduled queries configured through Fleet policies have no equivalent mechanism.
4. **Workaround is limited and fragile:** The only current workaround is creating a `logs-osquery_manager.result@custom` ingest pipeline with `set` processors. This is problematic because:
- The pipeline only executes at Elasticsearch index time. If events are routed to Logstash, Kafka, or other outputs first, the metadata is absent from those intermediate destinations.
- It is managed outside the integration policy, invisible in Fleet UI, and harder to maintain across deployments.
In contrast, `ecs_mapping` is applied by osquerybeat [*before* publishing](https://github.com/elastic/beats/blob/eff088939b9a4c2f1616a9b64eb5fe4a3352fea4/x-pack/osquerybeat/internal/pub/publisher.go#L227-L274), so the fields are present in the event regardless of output destination.
### What did you see?
Current stream template (`data_stream/result/agent/stream/stream.yml.hbs`):
```yaml
query: {{query}}
```
Current stream variables (`data_stream/result/manifest.yml`):
```yaml
vars:
- name: query
type: text
title: The query to be executed.
required: false
show_user: false
```
No `ecs_mapping` variable is defined or passed through to osquerybeat.
### Anything else?
**Proposed change:** Add a `type: yaml` variable for `ecs_mapping` to the result data stream manifest and stream template.
`data_stream/result/manifest.yml`:
```yaml
vars:
- name: query
type: text
title: The query to be executed.
required: false
show_user: false
- name: ecs_mapping
type: yaml
title: ECS Mapping
description: >-
Map osquery result fields to custom destination fields or inject static values.
Each key is a dotted destination field path. Use 'field' to copy an osquery
result column, or 'value' to inject a static constant. Example:
user.id:
field: "uid"
labels.environment:
value: "production"
required: false
show_user: false
```
`data_stream/result/agent/stream/stream.yml.hbs`:
```yaml
query: {{query}}
{{#if ecs_mapping}}
ecs_mapping: {{ecs_mapping}}
{{/if}}
```
This requires no changes to osquerybeat itself — the beat already parses and applies `ecs_mapping` from its [stream configuration](https://github.com/elastic/beats/blob/eff088939b9a4c2f1616a9b64eb5fe4a3352fea4/x-pack/osquerybeat/internal/config/config.go#L32-L40).
Contributor guide
Assessment
This issue has not been assessed yet.