elastic / elastic/integrations
[Elasticsearch]: querylog Agent stream does not decode NDJSON, so events stay unparsed when the x-pack template wins
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 182
Description
### Integration Name
Elasticsearch [packages/elasticsearch]
### Dataset Name
elasticsearch.querylog
### Summary
On ES 9.4+, query-log events shipped by the integration land as a JSON string in `message`. `elasticsearch.querylog.*` never populates, so the data view and dashboard are empty.
x-pack owns the destination template (`logs-elasticsearch.querylog@template`). The integration is only the shipper plus Kibana assets. The Agent stream is a plain `logfile` input. It never expands the NDJSON line. Parsing was left to `pipeline-json.yml` via `index.default_pipeline` on the Fleet template (priority 200). That template loses to the x-pack template (priority 240 after elastic/elasticsearch#158188, 250 before). The ingest pipeline never runs.
Filebeat already does this correctly. The `elasticsearch` querylog fileset uses `filestream` + `ndjson` (`target: ""`, `expand_keys: true`). Its ingest pipeline only sets `event.ingested` / `event.created`.
Related:
- Support: https://github.com/elastic/sdh-elasticsearch/issues/10223
- ES template priority: https://github.com/elastic/elasticsearch/pull/158188
- Original fileset: https://github.com/elastic/integrations/issues/13374
### What needs to change
**1. Decode on the Agent (required)**
`packages/elasticsearch/data_stream/querylog/agent/stream/log.yml.hbs`
Today it only has paths, tags, `allow_deprecated_use`, and `exclude_files`. Add JSON expansion equivalent to Filebeat. Smallest change that stays on `logfile` (the package policy template has no `filestream` input):
```yaml
json.keys_under_root: true
json.overwrite_keys: true
json.expand_keys: true
json.add_error_key: true
```
`expand_keys` matters. ES writes dotted ECS keys (`elasticsearch.querylog.type`, etc.).
Alternative: `decode_json_fields` on `message` with `target: ""`, `overwrite_keys: true`, `expand_keys: true`. Same outcome.
Do not switch this stream to `filestream` in this bugfix. That would need a new input on the package policy template and is a follow-up.
**2. Stop treating the ingest pipeline as load-bearing (optional but should do it)**
`packages/elasticsearch/data_stream/querylog/elasticsearch/ingest_pipeline/pipeline-json.yml` is doing the Agent’s job today. After shipper-side decode, keep `default.yml` for `event.ingested`, `event.created`, and `host.name` / `host.id` from `elasticsearch.node.*`.
Slim `pipeline-json.yml` so it is a no-op when `message` is already gone or already structured. Keep the `event.dataset != elasticsearch.querylog` drop if we still want that guard.
**3. Package metadata**
- Bump `packages/elasticsearch/manifest.yml` (current 1.21.1).
- Changelog entry, type `bugfix`.
- `querylog/manifest.yml` description still says “Collect … using log input”. Fine to leave, or mention NDJSON decode.
**4. Tests**
- Pipeline tests in `querylog/_dev/test/pipeline/` can stay. They still cover the remaining pipeline.
- System test (`querylog/_dev/test/system/test-default-config.yml`) should assert `elasticsearch.querylog.type` (or similar) is present on ingested docs.
- elastic-package stacks often do not install `logs-elasticsearch.querylog@template`. A passing system test there does not prove the 9.4+ conflict is fixed. The Agent-side decode is what makes the test representative.
### What not to change
Do not try to make the Fleet index template win on priority. Mappings, LogsDB, and DSL stay in x-pack. Do not put querylog parsing in `logs@custom`.
### Workaround until a release
On the Agent querylog stream, add the same `json.*` settings (or `pipeline` pointing at the installed `logs-elasticsearch.querylog-*` ingest pipeline). Request-level `pipeline` overrides `index.default_pipeline`.
### Expected result
With query logging enabled and the elasticsearch integration shipping `*_querylog.json`, documents in `logs-elasticsearch.querylog-*` have `elasticsearch.querylog.*` populated even when the x-pack template is the one that matched. The query analytics dashboard works without a manual pipeline override.
Contributor guide
Assessment
This issue has not been assessed yet.