elastic / elastic/integrations
aws_bedrock: invocation data stream causes 413 flush failures for streaming responses
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
The invocation data stream produces HTTP 413 errors when the Elastic Agent flushes events to Elasticsearch. There are two distinct problems with different root causes and different shipper paths.
**Problem 1: monitoring pipeline 413 (the observed symptom)**
The `"Permanent error: flush failed (413):"` error string is produced exclusively by the OTel ES exporter path:
- `"flush failed (413):"` comes from [`ErrorFlushFailed.Error()`](https://github.com/elastic/go-docappender/blob/a800a13021fde222e6d302e3cdd9cf316e6519a5/bulk_indexer.go#L729-L730) in go-docappender
- `"Permanent error:"` is the prefix added by [`consumererror.NewPermanent()`](https://github.com/open-telemetry/opentelemetry-collector/blob/4908404e59e544297b989a6961ee918b6f84b606/consumer/consumererror/permanent.go#L16-L21) when the status code is not in the retry list (413 is not retried — only 429 and 5xx are)
In elastic-agent 9.5.0, monitoring defaults to `OtelRuntimeManager` unconditionally ([`DefaultRuntimeManager = OtelRuntimeManager`](https://github.com/elastic/elastic-agent/blob/b64f023b38884f2d3c4717d8fe06085a7a9dcb61/internal/pkg/core/monitoring/config/config.go#L24)). The monitoring components run inside an OTel collector subprocess, which ships to Elasticsearch via the OTel ES exporter backed by go-docappender.
The Bedrock invocation logs go via a different path. AWS integrations use `ProcessRuntimeManager` and ship via libbeat's ES output, which does not use go-docappender and cannot produce this error format.
The 413 errors are from the monitoring pipeline failing to ship the `elastic_agent` dataset, not from the invocation data. The OTel collector logs the error to stdout, the agent captures it into its log file, the monitoring filestream picks it up and ships it as the `elastic_agent` dataset.
elastic-agent's [`ESToOTelConfig()`](https://github.com/elastic/elastic-agent/blob/b64f023b38884f2d3c4717d8fe06085a7a9dcb61/internal/pkg/otel/translate/output_elasticsearch.go#L58) overrides the OTel ES exporter's native defaults (5 MB bytes-based batch limit) with `sizer: "items"`, `max_size: 1600`, and no compression. Because the sizer is `"items"` rather than `"bytes"`, the byte-ceiling code path in [`newSyncBulkIndexer()`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/89e43555904cd97c2d36605347c5d5237b1bdc8c/exporter/elasticsearchexporter/bulkindexer.go#L143-L160) sets `maxFlushBytes = 0`, which disables per-item byte checking entirely. Up to 1,600 monitoring events are sent as a single uncompressed NDJSON bulk request with no byte cap. When that payload exceeds `http.max_content_length`, ES returns 413.
There is also a feedback loop. HTTP 413 is logged by the OTel collector to `elastic-otel-collector-*.ndjson`. The monitoring filestream reads that file and includes the error line in the next monitoring batch. The existing [drop filters](https://github.com/elastic/elastic-agent/blob/b64f023b38884f2d3c4717d8fe06085a7a9dcb61/internal/pkg/agent/application/monitoring/component/v1_monitor.go#L1034-L1042) do not catch it: one matches on `otelcol.component.id` ending in `_agent-component/monitoring`, which does not match the exporter component ID on these error logs; the other matches specific metric-reporting message strings and does not match `"Permanent error: flush failed (413):"`. The loop is self-reinforcing but bounded — each 413 adds a fixed number of new monitoring events rather than growing without limit.
The relevant knob for this path is `flush::bytes` (or enabling compression) in the OTel ES exporter config, not `bulk_max_size`.
**Problem 2: large invocation documents silently dropped by libbeat**
Large Bedrock invocation documents can cause data loss via libbeat, though less visibly. libbeat handles a batch-level 413 by progressively halving the batch until it isolates a single event that still fails, then drops that document with a warning log, without producing the `"Permanent error:"` string.
The ingest pipeline truncates or removes `gen_ai.prompt`, `gen_ai.completion`, `input_body_json`, and `output_body_json`. `event.original` is handled by the Fleet final pipeline when `preserve_original_event` is false. The field that bypasses all truncation is `aws_bedrock.invocation.messages` — the normalized conversation history. The [pipeline scripts](https://github.com/elastic/integrations/blob/8acd9132737687b07b1055ade4efc6c931e2133e/packages/aws_bedrock/data_stream/invocation/elasticsearch/ingest_pipeline/default.yml#L241) normalise the message structure but do not bound its size. For multi-turn Bedrock conversations with large per-turn content this field is unbounded and is the main candidate for oversized invocation documents reaching libbeat's drop path.
Additionally, the `max_bytes` stream parameter documented in the manifest only applies to non-JSON log files. Because Bedrock invocation logs are JSON, this limit is never evaluated. The 10 MiB default gives false assurance that there is an effective per-event cap. The ingest pipeline's truncation guards run inside Elasticsearch after the bulk HTTP request is accepted and cannot prevent a rejection at the HTTP layer.
Contributor guide
Research direction
Start by tracing internal/pkg/otel/translate/output_elasticsearch.go and internal/pkg/agent/application/monitoring/component/v1_monitor.go for the monitoring path, then inspect packages/aws_bedrock/data_stream/invocation/elasticsearch/ingest_pipeline/default.yml for oversized invocation fields. Reproduce the 413 behavior and determine which path is in scope; done should prevent the identified request failures or document loss without introducing the monitoring feedback loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, elasticsearch
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100