elastic / elastic/integrations
[panw_cortex_xdr] Consolidate the API data streams onto a single CEL input
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
## The problem (user's view)
Pointing this integration at one Cortex XDR tenant makes you enter the same credentials more than once. The two API-based data streams are on different Elastic Agent inputs: `alerts` runs on CEL (the current v2 API), while `incidents` is still on HTTPJSON and was never migrated. Because each input carries its own copy of the connection config, a user has to supply the same API domain, API token, and token ID separately for the CEL stream and the HTTPJSON stream.
On top of that, the `alerts` stream still ships a deprecated HTTPJSON block for the old v1 API alongside its CEL v2 path, so there's dead weight sitting in the package that users can still stumble into.
None of this is required by Cortex XDR. It's the usual story: alerts got moved to CEL, incidents didn't, and the deprecated path was left behind.
## Where we are
Three data streams, three different collection methods (v2.7.0):
- `alerts` → **CEL** (v2 API), plus a **deprecated HTTPJSON** v1 block still shipping alongside it
- `incidents` → **HTTPJSON** only, never migrated
- `event` → **GCS**
The API side is the issue. `alerts` and `incidents` hit the same tenant API but sit on different inputs, and the leftover v1 HTTPJSON on `alerts` is egacy.
## Proposal
Collapse the API side onto one CEL input:
1. Migrate `incidents` to CEL.
2. Retire the deprecated `alerts` v1 HTTPJSON block.
After that, everything that talks to the Cortex XDR API runs through CEL, and the tenant connection is configured once.
## Leave `event`/GCS alone
The `event` stream is deliberately out of scope. It doesn't hit the Cortex API — Cortex XDR's Event Forwarding feature pushes raw endpoint events into a GCS bucket, and the Elastic GCS input reads that bucket using a Google service-account key. It's a genuinely different transport with different credentials, GCS is Palo Alto's only native export destination for it, so it can't and shouldn't fold into CEL.
## Why it's worth doing
- Credentials entered once for the API side instead of twice.
- One input to explain in the docs and in support, and the confusing deprecated v1 path is gone.
- CEL-only improvements (state persistence, dedup, error handling) cover both API streams. There's an open HTTPJSON cursor bug (#19392) that retiring/replacing the HTTPJSON path helps put to bed.
## What to watch out for
The upgrade is the risky part. Switching `incidents` from HTTPJSON to CEL can reset its cursor and cause a gap or duplicate events at the cutover, so we need a cursor-continuity plan and tests that prove no gaps or duplicates before release. Removing the deprecated `alerts` v1 block needs a clear deprecation path and changelog note for anyone still on it.
## Rough breakdown
1. Migrate `incidents` to CEL + tests.
2. Retire the deprecated `alerts` v1 HTTPJSON block with a documented deprecation path.
3. Prove the upgrade is clean: no gaps, no duplicates at cutover.
4. Update docs and changelog; leave `event`/GCS untouched.
Contributor guide
Research direction
Start with the panw_cortex_xdr integration's alerts and incidents stream definitions, comparing their current CEL and HTTPJSON inputs. Review issue #19392 and the existing upgrade behavior before planning cursor-continuity tests. Done means incidents uses CEL, the deprecated alerts v1 HTTPJSON block is retired with a deprecation note, cutover has no gaps or duplicates, and event/GCS remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- handlebars
- Domain
- api, security
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100