elastic / elastic/integrations

[Prisma Access] Add S3 ingestion path for Strata Logging Service JSON logs

Open
#18,054 5 comments 0 reactions 0 assignees View on GitHub
enhancement Integration:prisma_access needs:triage Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

Palo Alto Prisma Access [prisma_access]

### Dataset Name

prisma_access.event (new input type for existing data stream)

### Integration Version

1.7.0

### Agent Version

9.3.1

### OS Version and Architecture

N/A — architecture-level feature request

### User Goal

Add an alternative ingestion path to the Prisma Access integration that reads JSON logs from an S3 bucket, using the `aws-s3` input. Palo Alto's [Strata Logging Service supports forwarding logs to AWS S3](https://docs.paloaltonetworks.com/strata-logging-service/administration/forward-logs/forward-logs-to-aws-s3) as batched, Snappy-compressed JSON files.

This S3-based path simplifies the ingestion architecture. The current TCP/CEF syslog path requires deploying multiple Elastic Agents behind a load balancer to handle high event rates, adding operational complexity. S3 provides natural buffering that decouples the ingestion rate from the processing rate and eliminates the need for load-balanced agent fleets.

### Existing Features

The Prisma Access integration currently only supports ingestion via TCP syslog (CEF format). While this path can be scaled to high throughput by adding agents and increasing worker counts, doing so requires a complex architecture: multiple Elastic Agents behind a load balancer, careful tuning of worker concurrency, and significant CPU resources for Filebeat's `decode_cef` processing.

An S3-based ingestion path would be significantly simpler to operate and scale.

### What did you see?

The TCP/CEF syslog path works but requires considerable architectural effort to scale — multiple agents, load balancing, and tuning. The S3 path is a cleaner solution because it eliminates the need for load-balanced agent fleets entirely and leverages S3's built-in buffering and durability.

### Implementation Details

#### Prerequisites

1. **Snappy decompression support in the `aws-s3` input (Beats).**
Strata Logging Service compresses S3 objects using [Snappy](https://github.com/google/snappy). The Elastic `aws-s3` input currently only supports gzip decompression. Snappy support must be added.
- A proof-of-concept branch exists: [`andrewkroh/beats@add-snappy-decompression-support`](https://github.com/andrewkroh/beats/tree/add-snappy-decompression-support)
- This should be contributed upstream to `elastic/beats` as a standalone PR.

2. **Access to Palo Alto Strata Logging Service.**
We need to verify the exact JSON field schema produced by the S3 forwarding destination. The Palo Alto documentation confirms the output is JSON but does not publish a complete field schema. Sample data from Strata is required to:
- Confirm field names (are they the same as CEF extension key names, or different?).
- Determine whether the existing ingest pipeline can be adapted or a new pipeline is needed.
- Build pipeline test fixtures.

#### Integration Changes

- Add a new input variant to the Prisma Access integration using the `aws-s3` input type, alongside the existing TCP input.
- Write or adapt an ingest pipeline that handles the JSON field format from Strata Logging Service (the current pipeline expects CEF-decoded fields under `cef.extensions.*`).
- Add pipeline tests with representative sample data from Strata.

### Anything else?

**Palo Alto documentation reference:**
- [Forward Logs to AWS S3](https://docs.paloaltonetworks.com/strata-logging-service/administration/forward-logs/forward-logs-to-aws-s3) — confirms Snappy compression and JSON format: *"Strata Logging Service batch logs based on either a size limit of 1000 lines or a time limit of 10 seconds, whichever is reached first. Strata Logging Service compresses the log file using Snappy and forwards it to S3 bucket in JSON format."*

**Related alternatives considered:**
- **Strata → Security Lake (S3 Parquet/OCSF):** OCSF is a different schema from CEF, requiring a full pipeline rewrite. Risk of lossy translation for vendor-specific PanOS fields. Higher complexity.
- **Optimize existing TCP/CEF path:** Addresses symptoms but the architecture remains complex to operate at scale.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.