elastic / elastic/integrations

[panw_cortex_xdr] Consolidate the API data streams onto a single CEL input

Open
#20,288 3 comments 0 reactions 0 assignees View on GitHub
enhancement Integration:panw_cortex_xdr Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

## The problem (user's view)

Pointing this integration at one Cortex XDR tenant makes you enter the same credentials more than once. The two API-based data streams are on different Elastic Agent inputs: `alerts` runs on CEL (the current v2 API), while `incidents` is still on HTTPJSON and was never migrated. Because each input carries its own copy of the connection config, a user has to supply the same API domain, API token, and token ID separately for the CEL stream and the HTTPJSON stream.

On top of that, the `alerts` stream still ships a deprecated HTTPJSON block for the old v1 API alongside its CEL v2 path, so there's dead weight sitting in the package that users can still stumble into.

None of this is required by Cortex XDR. It's the usual story: alerts got moved to CEL, incidents didn't, and the deprecated path was left behind.

## Where we are

Three data streams, three different collection methods (v2.7.0):

- `alerts` → **CEL** (v2 API), plus a **deprecated HTTPJSON** v1 block still shipping alongside it
- `incidents` → **HTTPJSON** only, never migrated
- `event` → **GCS**

The API side is the issue. `alerts` and `incidents` hit the same tenant API but sit on different inputs, and the leftover v1 HTTPJSON on `alerts` is egacy.

## Proposal

Collapse the API side onto one CEL input:

1. Migrate `incidents` to CEL.
2. Retire the deprecated `alerts` v1 HTTPJSON block.

After that, everything that talks to the Cortex XDR API runs through CEL, and the tenant connection is configured once.

## Leave `event`/GCS alone

The `event` stream is deliberately out of scope. It doesn't hit the Cortex API — Cortex XDR's Event Forwarding feature pushes raw endpoint events into a GCS bucket, and the Elastic GCS input reads that bucket using a Google service-account key. It's a genuinely different transport with different credentials, GCS is Palo Alto's only native export destination for it, so it can't and shouldn't fold into CEL.

## Why it's worth doing

- Credentials entered once for the API side instead of twice.
- One input to explain in the docs and in support, and the confusing deprecated v1 path is gone.
- CEL-only improvements (state persistence, dedup, error handling) cover both API streams. There's an open HTTPJSON cursor bug (#19392) that retiring/replacing the HTTPJSON path helps put to bed.

## What to watch out for

The upgrade is the risky part. Switching `incidents` from HTTPJSON to CEL can reset its cursor and cause a gap or duplicate events at the cutover, so we need a cursor-continuity plan and tests that prove no gaps or duplicates before release. Removing the deprecated `alerts` v1 block needs a clear deprecation path and changelog note for anyone still on it.

## Rough breakdown

1. Migrate `incidents` to CEL + tests.
2. Retire the deprecated `alerts` v1 HTTPJSON block with a documented deprecation path.
3. Prove the upgrade is clean: no gaps, no duplicates at cutover.
4. Update docs and changelog; leave `event`/GCS untouched.

Contributor guide

Open the contributing guide

Research direction

Start with the panw_cortex_xdr integration's alerts and incidents stream definitions, comparing their current CEL and HTTPJSON inputs. Review issue #19392 and the existing upgrade behavior before planning cursor-continuity tests. Done means incidents uses CEL, the deprecated alerts v1 HTTPJSON block is retired with a deprecation note, cutover has no gaps or duplicates, and event/GCS remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
handlebars
Domain
api, security
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.