elastic / elastic/integrations
[Azure Logs]: Optimize Azure managed ingest pipelines by replacing expensive Grok with Dissect where feasible
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
Azure Logs [packages/azure]
### Dataset Name
_No response_
### Integration Version
1.36.1
### Agent Version
9.3.3
### OS Version and Architecture
Linux 5.15.0-1086-azure on some nodes and 5.15.0-1037-azure on others, amd64
### User Goal
Replace or reduce Grok (regex) processors in Azure managed ingest pipelines with [Dissect](https://www.elastic.co/docs/reference/enrich-processor/dissect-processor) where message formats allow fixed-token parsing, to lower CPU/heap cost and backtracking.
### Existing Features
- Guidance to use `{type}-{dataset}@custom` [pipeline](https://www.elastic.co/docs/reference/fleet/data-streams-pipeline-tutorial) hook to add pre-drop/remove or pre-dissect, or reroute to a custom pipeline.
- Scaled ingest nodes; considered batch size tuning.
### What did you see?
- Heavy [Azure logs](https://www.elastic.co/docs/reference/integrations/azure) ingestion via Fleet-managed pipelines (e.g., logs-azure.platformlogs-1.36.1-azure-shared-pipeline) causing high ingest node heap, GC pressure, and indexing rejections.
- User observed strong correlation between Azure pipelines’ Grok usage and heap spikes; requests performance optimization in parsing.
- Elastic Cloud Platform (ESS); Elasticsearch version 9.3.3 in case data; Azure integration managed pipelines variant ~1.36.1 observed.
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.