elastic / elastic/integrations

[Azure Logs]: Optimize Azure managed ingest pipelines by replacing expensive Grok with Dissect where feasible

Open
#18,826 1 comment 0 reactions 0 assignees View on GitHub
Integration:azure needs:triage Team:obs-ds-hosted-services
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

Azure Logs [packages/azure]

### Dataset Name

_No response_

### Integration Version

1.36.1

### Agent Version

9.3.3

### OS Version and Architecture

Linux 5.15.0-1086-azure on some nodes and 5.15.0-1037-azure on others, amd64

### User Goal

Replace or reduce Grok (regex) processors in Azure managed ingest pipelines with [Dissect](https://www.elastic.co/docs/reference/enrich-processor/dissect-processor) where message formats allow fixed-token parsing, to lower CPU/heap cost and backtracking.

### Existing Features

- Guidance to use `{type}-{dataset}@custom` [pipeline](https://www.elastic.co/docs/reference/fleet/data-streams-pipeline-tutorial) hook to add pre-drop/remove or pre-dissect, or reroute to a custom pipeline.
- Scaled ingest nodes; considered batch size tuning.

### What did you see?

- Heavy [Azure logs](https://www.elastic.co/docs/reference/integrations/azure) ingestion via Fleet-managed pipelines (e.g., logs-azure.platformlogs-1.36.1-azure-shared-pipeline) causing high ingest node heap, GC pressure, and indexing rejections.
- User observed strong correlation between Azure pipelines’ Grok usage and heap spikes; requests performance optimization in parsing.
- Elastic Cloud Platform (ESS); Elasticsearch version 9.3.3 in case data; Azure integration managed pipelines variant ~1.36.1 observed.

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.