elastic / elastic/integrations

[docs-patrol] Document AWS VPC Flow unsupported-format pipeline_error behavior

Open
#19,165 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

Recent code changes in the repository have introduced documentation drift. The following change needs a corresponding documentation update.

## Changes Requiring Documentation Updates

### 1. AWS VPC Flow now emits pipeline errors for unsupported plain-text formats

**Commit(s):**
- [`ddd75707f42fb6fbea02af51f4fec692f842b5a7`](https://github.com/elastic/integrations/commit/ddd75707f42fb6fbea02af51f4fec692f842b5a7)

**What changed:**
- The VPC Flow ingest pipeline added a `fail` processor for records whose token count does not match supported layouts and where parsing did not populate `aws.vpcflow.srcaddr`.
- This changed behavior from silently indexing unsupported formats unparsed to surfacing them as pipeline errors.

**Evidence:**
- `packages/aws/data_stream/vpcflow/elasticsearch/ingest_pipeline/default.yml:72-77` adds:
- `fail` on `ctx?._temp_?.message_token_count != null && ctx?.aws?.vpcflow?.srcaddr == null`
- message: `Unsupported VPC Flow Log format... Use a custom ingest pipeline to parse this format.`
- `packages/aws/changelog.yml:12-16` documents this as version `6.16.0` behavior change.

**Documentation impact:**
- `packages/aws/docs/vpcflow.md` documents supported formats (`:51-62`) and states only that Parquet is unsupported (`:86`), but does not explain the new unsupported plain-text failure behavior.
- Users now need explicit guidance that unsupported plain-text formats generate `pipeline_error` events and that custom ingest pipelines are required for those layouts.

## Suggested Actions

- [ ] Update `packages/aws/docs/vpcflow.md` to document that unsupported plain-text VPC Flow layouts now trigger pipeline errors instead of being silently indexed.
- [ ] Add a short troubleshooting note describing expected error behavior and recommending custom ingest pipelines for unsupported formats.
- [ ] If applicable, add an example of an unsupported custom layout symptom (`pipeline_error` with unsupported format message) so users can recognize this condition quickly.

> [!NOTE]
>
> 🔒 Integrity filter blocked 37 items
>
> The following items were blocked because they don't meet the GitHub integrity level.
>
> - [#18066](https://github.com/elastic/integrations/issues/18066) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18164](https://github.com/elastic/integrations/issues/18164) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#19124](https://github.com/elastic/integrations/pull/19124) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18498](https://github.com/elastic/integrations/pull/18498) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#17506](https://github.com/elastic/integrations/pull/17506) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#19140](https://github.com/elastic/integrations/pull/19140) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18671](https://github.com/elastic/integrations/pull/18671) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#19034](https://github.com/elastic/integrations/pull/19034) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#19061](https://github.com/elastic/integrations/pull/19061) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18621](https://github.com/elastic/integrations/pull/18621) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18746](https://github.com/elastic/integrations/pull/18746) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18955](https://github.com/elastic/integrations/pull/18955) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18944](https://github.com/elastic/integrations/pull/18944) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18765](https://github.com/elastic/integrations/pull/18765) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18951](https://github.com/elastic/integrations/pull/18951) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#18417](https://github.com/elastic/integrations/pull/18417) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - ... and 21 more items
>
> To allow these resources, lower `min-integrity` in your GitHub frontmatter:
>
> ```yaml
> tools:
> github:
> min-integrity: approved # merged | approved | unapproved | none
> ```
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Docs Patrol](https://github.com/elastic/integrations/actions/runs/26283774767)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on May 29, 2026, 11:06 AM UTC

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.