elastic / elastic/integrations

[Elasticsearch]: querylog Agent stream does not decode NDJSON, so events stay unparsed when the x-pack template wins

Open
#20,985 4 comments 0 reactions 1 assignee Claimed by @consulthys View on GitHub
bug Feature:Stack Monitoring Integration:elasticsearch Team:Stack Monitoring
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 18h
Merged PRs (30d)
182

Description

### Integration Name

Elasticsearch [packages/elasticsearch]

### Dataset Name

elasticsearch.querylog

### Summary

On ES 9.4+, query-log events shipped by the integration land as a JSON string in `message`. `elasticsearch.querylog.*` never populates, so the data view and dashboard are empty.

x-pack owns the destination template (`logs-elasticsearch.querylog@template`). The integration is only the shipper plus Kibana assets. The Agent stream is a plain `logfile` input. It never expands the NDJSON line. Parsing was left to `pipeline-json.yml` via `index.default_pipeline` on the Fleet template (priority 200). That template loses to the x-pack template (priority 240 after elastic/elasticsearch#158188, 250 before). The ingest pipeline never runs.

Filebeat already does this correctly. The `elasticsearch` querylog fileset uses `filestream` + `ndjson` (`target: ""`, `expand_keys: true`). Its ingest pipeline only sets `event.ingested` / `event.created`.

Related:
- Support: https://github.com/elastic/sdh-elasticsearch/issues/10223
- ES template priority: https://github.com/elastic/elasticsearch/pull/158188
- Original fileset: https://github.com/elastic/integrations/issues/13374

### What needs to change

**1. Decode on the Agent (required)**

`packages/elasticsearch/data_stream/querylog/agent/stream/log.yml.hbs`

Today it only has paths, tags, `allow_deprecated_use`, and `exclude_files`. Add JSON expansion equivalent to Filebeat. Smallest change that stays on `logfile` (the package policy template has no `filestream` input):

```yaml
json.keys_under_root: true
json.overwrite_keys: true
json.expand_keys: true
json.add_error_key: true
```

`expand_keys` matters. ES writes dotted ECS keys (`elasticsearch.querylog.type`, etc.).

Alternative: `decode_json_fields` on `message` with `target: ""`, `overwrite_keys: true`, `expand_keys: true`. Same outcome.

Do not switch this stream to `filestream` in this bugfix. That would need a new input on the package policy template and is a follow-up.

**2. Stop treating the ingest pipeline as load-bearing (optional but should do it)**

`packages/elasticsearch/data_stream/querylog/elasticsearch/ingest_pipeline/pipeline-json.yml` is doing the Agent’s job today. After shipper-side decode, keep `default.yml` for `event.ingested`, `event.created`, and `host.name` / `host.id` from `elasticsearch.node.*`.

Slim `pipeline-json.yml` so it is a no-op when `message` is already gone or already structured. Keep the `event.dataset != elasticsearch.querylog` drop if we still want that guard.

**3. Package metadata**

- Bump `packages/elasticsearch/manifest.yml` (current 1.21.1).
- Changelog entry, type `bugfix`.
- `querylog/manifest.yml` description still says “Collect … using log input”. Fine to leave, or mention NDJSON decode.

**4. Tests**

- Pipeline tests in `querylog/_dev/test/pipeline/` can stay. They still cover the remaining pipeline.
- System test (`querylog/_dev/test/system/test-default-config.yml`) should assert `elasticsearch.querylog.type` (or similar) is present on ingested docs.
- elastic-package stacks often do not install `logs-elasticsearch.querylog@template`. A passing system test there does not prove the 9.4+ conflict is fixed. The Agent-side decode is what makes the test representative.

### What not to change

Do not try to make the Fleet index template win on priority. Mappings, LogsDB, and DSL stay in x-pack. Do not put querylog parsing in `logs@custom`.

### Workaround until a release

On the Agent querylog stream, add the same `json.*` settings (or `pipeline` pointing at the installed `logs-elasticsearch.querylog-*` ingest pipeline). Request-level `pipeline` overrides `index.default_pipeline`.

### Expected result

With query logging enabled and the elasticsearch integration shipping `*_querylog.json`, documents in `logs-elasticsearch.querylog-*` have `elasticsearch.querylog.*` populated even when the x-pack template is the one that matched. The query analytics dashboard works without a manual pipeline override.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.