Azure Logs Ingestion API output doesn't limit request payload size
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
**Describe the bug**
Azure Monitor's Logs Ingestion API has a maximum payload size of 1MiB. The `azure_logs_ingestion` output sends one http request per flushed chunk. A chunk serialized to json may exceed 1MiB, resulting in persistent 413 responses.
Enabling compression reduces the chances of the maximum payload size being exceeded, but does not eliminate it.
**To Reproduce**
1. In Azure, create a log analytics workspace, direct data collection rule and a service principal for RBAC.
2. Run Fluent Bit with the below config.
3. Monitor Fluent Bit output.
```yaml
service:
flush: 10
grace: 2
log_level: warn
pipeline:
inputs:
- name: dummy
tag: two_mb_uncompressed
dummy: '{"SeverityText":"Info","Body":"Fluent Bit full chunk sent uncompressed to Azure Monitor","Attributes":{"repro":"azure-logs-ingestion-2mb-uncompressed"}}'
samples: 300
copies: 100
rate: 100
fixed_timestamp: true
flush_on_startup: true
outputs:
- name: azure_logs_ingestion
match: two_mb_uncompressed
table_name: MyFluentBitLogs
dce_url: ${DCE_URL}
dcr_id: ${DCR_ID}
tenant_id: ${TENANT_ID}
client_id: ${CLIENT_ID}
client_secret: ${CLIENT_SECRET}
time_key: TimeGenerated
time_generated: true
compress: false
retry_limit: no_retries
```
**Expected behavior**
Given individual log records with a json representation under 1MiB (possibly compressed), the `azure_logs_ingestion` output plugin would split a chunk into 1-N segments that each have a payload size under the limit.
If one or more of the requests to the Logs Ingestion API fail, the plugin returns `FLB_RETRY` or `FLB_ERROR` to FB core. If retries are enabled, some log records might be ingested multiple times in the case of transient failures.
**Screenshots**
```
Fluent Bit v5.1.2
* Copyright (C) 2015-2026 The Fluent Bit Authors
* Fluent Bit is a CNCF graduated project under the Fluent organization
* https://fluentbit.io
______ _ _ ______ _ _ _____ __
| ___| | | | | ___ (_) | | ___|/ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ _|___ \ `| |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / \ \ | |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V //\__/ /_| |_
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ \____(_)___/
[2026/09/14 22:30:43.368] [ warn] [output:azure_logs_ingestion:azure_logs_ingestion.0] http_status=413:
{"error":{"code":"ContentLengthLimitExceeded","message":"Maximum allowed content length: 1048576 bytes (1 MB). Provided content length: 2646001 bytes."}}
[2026/09/14 22:30:47.625] [ warn] [output:azure_logs_ingestion:azure_logs_ingestion.0] http_status=413:
{"error":{"code":"ContentLengthLimitExceeded","message":"Maximum allowed content length: 1048576 bytes (1 MB). Provided content length: 2646001 bytes."}}
```
**Your Environment**
* Version used: v5.1.2 (homebrew)
* Configuration:
* Environment name and version (e.g. Kubernetes? What version?):
* Server type and version:
* Operating System and version: macOS Tahoe (Darwin Kernel Version 25.6.0)
* Filters and plugins: `dummy`, `azure_logs_ingestion`
**Additional context**
GitHub's observability team spotted this bug while we evaluated significantly increasing Fluent Bit's average chunk size to reduce request overhead. Ideally request payloads are slightly under 1MiB. This bug hasn't impacted us yet.
https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/service-limits#logs-ingestion-api
dotnet client's segmentation of the incoming log stream:
https://github.com/Azure/azure-sdk-for-net/blob/dc8ee2a68d951f812b1413bb74f659b65edc1d3c/sdk/monitor/Azure.Monitor.Ingestion/src/LogsIngestionClient.cs#L124-L125
Related: https://github.com/fluent/fluent-bit/issues/1938
Contributor guide
Research direction
Start with the azure_logs_ingestion output and reproduce the issue using the YAML configuration and dummy input described here, checking the serialized request size and the 413 response. Trace how one flushed chunk becomes an HTTP request, including compression and retry handling. Done means records are split into 1-N requests under Azure's 1MiB limit and failures return FLB_RETRY or FLB_ERROR as specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, c
- Domain
- api, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100