elastic / elastic/integrations
[bug-hunter] unifiedlogs custom dataset is ignored due to hardcoded event.dataset
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Impact
Users who configure a custom dataset for the `unifiedlogs` input (for example `elastic_agent.test`) still get events with `event.dataset: unifiedlogs.log`. This creates deterministic field mismatch (`data_stream.dataset != event.dataset`) and breaks dataset-based filters, dashboards, and detection/content that relies on `event.dataset`.
## Reproduction Steps
1. Save and run this script from repo root:
```python
import yaml
from copy import deepcopy
pipeline_path = 'packages/unifiedlogs/elasticsearch/ingest_pipeline/default.yml'
manifest_path = 'packages/unifiedlogs/manifest.yml'
input_tpl_path = 'packages/unifiedlogs/agent/input/unifiedlogs.yml.hbs'
with open(pipeline_path, 'r', encoding='utf-8') as f:
pipeline = yaml.safe_load(f)
with open(manifest_path, 'r', encoding='utf-8') as f:
manifest = yaml.safe_load(f)
vars_list = manifest['policy_templates'][0]['vars']
assert any(v.get('name') == 'data_stream.dataset' for v in vars_list)
with open(input_tpl_path, 'r', encoding='utf-8') as f:
tpl = f.read()
assert 'dataset: \{\{data_stream.dataset}}' in tpl
doc = {
'data_stream': {'dataset': 'elastic_agent.test'},
'message': '{"eventMessage":"hello"}'
}
for p in pipeline.get('processors', []):
if 'set' in p:
s = p['set']
field = s.get('field')
value = s.get('value')
if field and value is not None:
cur = doc
parts = field.split('.')
for part in parts[:-1]:
if part not in cur or not isinstance(cur[part], dict):
cur[part] = {}
cur = cur[part]
cur[parts[-1]] = deepcopy(value)
expected = doc['data_stream']['dataset']
actual = doc.get('event', {}).get('dataset')
print(f'data_stream.dataset={expected}')
print(f'event.dataset={actual}')
if actual != expected:
print('BUG REPRODUCED: event.dataset is hardcoded and does not follow configured dataset')
raise SystemExit(1)
```
2. Run:
```bash
python /tmp/gh-aw/agent/repro_unifiedlogs_dataset.py
```
## Expected vs Actual
**Expected:** `event.dataset` should match the configured `data_stream.dataset` (e.g. `elastic_agent.test`).
**Actual:** `event.dataset` is always `unifiedlogs.log`.
Observed output:
```text
data_stream.dataset=elastic_agent.test
event.dataset=unifiedlogs.log
BUG REPRODUCED: event.dataset is hardcoded and does not follow configured dataset
```
## Failing Test
The reproduction script above is a minimal failing test (exits non-zero when mismatch is present).
## Evidence
- `packages/unifiedlogs/elasticsearch/ingest_pipeline/default.yml:23-26` unconditionally sets:
- `field: event.dataset`
- `value: unifiedlogs.log`
- `packages/unifiedlogs/agent/input/unifiedlogs.yml.hbs:2-3` wires user-configurable `data_stream.dataset`.
- `packages/unifiedlogs/manifest.yml:27-35` exposes `data_stream.dataset` as required user var.
- Commit `c2f386e55078c0ae340661793d2d0e6613b3780d` introduced dataset configurability, but ingest pipeline still hardcodes `event.dataset`.
> [!NOTE]
>
> 🔒 Integrity filtering filtered 2 items
>
> Integrity filtering activated and filtered the following items during workflow execution.
> This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.
>
> - issue:elastic/integrations#unknown (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - issue:#unknown (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
>
>
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Bug Hunter](https://github.com/elastic/integrations/actions/runs/23846058797)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Apr 8, 2026, 11:38 AM UTC
Contributor guide
Assessment
This issue has not been assessed yet.