elastic / elastic/integrations

[bug-hunter] unifiedlogs custom dataset is ignored due to hardcoded event.dataset

Open
#18,180 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Impact
Users who configure a custom dataset for the `unifiedlogs` input (for example `elastic_agent.test`) still get events with `event.dataset: unifiedlogs.log`. This creates deterministic field mismatch (`data_stream.dataset != event.dataset`) and breaks dataset-based filters, dashboards, and detection/content that relies on `event.dataset`.

## Reproduction Steps
1. Save and run this script from repo root:

```python
import yaml
from copy import deepcopy

pipeline_path = 'packages/unifiedlogs/elasticsearch/ingest_pipeline/default.yml'
manifest_path = 'packages/unifiedlogs/manifest.yml'
input_tpl_path = 'packages/unifiedlogs/agent/input/unifiedlogs.yml.hbs'

with open(pipeline_path, 'r', encoding='utf-8') as f:
pipeline = yaml.safe_load(f)

with open(manifest_path, 'r', encoding='utf-8') as f:
manifest = yaml.safe_load(f)

vars_list = manifest['policy_templates'][0]['vars']
assert any(v.get('name') == 'data_stream.dataset' for v in vars_list)

with open(input_tpl_path, 'r', encoding='utf-8') as f:
tpl = f.read()
assert 'dataset: \{\{data_stream.dataset}}' in tpl

doc = {
'data_stream': {'dataset': 'elastic_agent.test'},
'message': '{"eventMessage":"hello"}'
}

for p in pipeline.get('processors', []):
if 'set' in p:
s = p['set']
field = s.get('field')
value = s.get('value')
if field and value is not None:
cur = doc
parts = field.split('.')
for part in parts[:-1]:
if part not in cur or not isinstance(cur[part], dict):
cur[part] = {}
cur = cur[part]
cur[parts[-1]] = deepcopy(value)

expected = doc['data_stream']['dataset']
actual = doc.get('event', {}).get('dataset')
print(f'data_stream.dataset={expected}')
print(f'event.dataset={actual}')
if actual != expected:
print('BUG REPRODUCED: event.dataset is hardcoded and does not follow configured dataset')
raise SystemExit(1)
```

2. Run:

```bash
python /tmp/gh-aw/agent/repro_unifiedlogs_dataset.py
```

## Expected vs Actual
**Expected:** `event.dataset` should match the configured `data_stream.dataset` (e.g. `elastic_agent.test`).

**Actual:** `event.dataset` is always `unifiedlogs.log`.

Observed output:

```text
data_stream.dataset=elastic_agent.test
event.dataset=unifiedlogs.log
BUG REPRODUCED: event.dataset is hardcoded and does not follow configured dataset
```

## Failing Test
The reproduction script above is a minimal failing test (exits non-zero when mismatch is present).

## Evidence
- `packages/unifiedlogs/elasticsearch/ingest_pipeline/default.yml:23-26` unconditionally sets:
- `field: event.dataset`
- `value: unifiedlogs.log`
- `packages/unifiedlogs/agent/input/unifiedlogs.yml.hbs:2-3` wires user-configurable `data_stream.dataset`.
- `packages/unifiedlogs/manifest.yml:27-35` exposes `data_stream.dataset` as required user var.
- Commit `c2f386e55078c0ae340661793d2d0e6613b3780d` introduced dataset configurability, but ingest pipeline still hardcodes `event.dataset`.

> [!NOTE]
>
> 🔒 Integrity filtering filtered 2 items
>
> Integrity filtering activated and filtered the following items during workflow execution.
> This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.
>
> - issue:elastic/integrations#unknown (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - issue:#unknown (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Bug Hunter](https://github.com/elastic/integrations/actions/runs/23846058797)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Apr 8, 2026, 11:38 AM UTC

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.