elastic / elastic/integrations

[Google Workspace] Consolidate CEL and httpjson inputs

Open
#14,813 4 comments 0 reactions 0 assignees View on GitHub
Integration:google_workspace Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## The problem (user's view)

Setting up Google Workspace is more work than it should be, and the reason is invisible to the user. The integration is split across two Elastic Agent inputs, HTTPJSON and CEL, and that split leaks into the setup. Someone has to enter the same Google service-account JWT, delegated admin account, and API host twice, once for the HTTPJSON input group and once for the CEL group, because each input carries its own copy of the auth config. Then they have to figure out which input serves which event type and toggle streams in two separate places.

Nothing about Google Workspace requires this. It's just history. The older streams landed on HTTPJSON, the newer ones on CEL, and the seam between them is now something every user has to deal with.

## Where we are

The split is large and one-sided. Of the 22 data streams, 14 are still on HTTPJSON and only 8 are on CEL (v3.5.0), and unlike some other integrations none of the HTTPJSON streams have been moved yet. So this is a real chunk of work, not a finishing touch.

- **HTTPJSON (14):** access_transparency, admin, alert, context_aware_access, device, drive, gcp, group_enterprise, groups, login, rules, saml, token, user_accounts
- **CEL (8):** calendar, chat, chrome, data_studio, gmail, keep, meet, vault

Most of the HTTPJSON streams are Reports API pulls with a shared shape, so they should migrate as a group rather than one painful stream at a time. Gmail is worth a callout: it's CEL already but BigQuery-based, so its config surface differs from the Reporting-API streams and shouldn't be assumed to fit a single unified input without a look.

## Proposal

Move the HTTPJSON streams onto CEL, make CEL the primary input, and hide HTTPJSON for new users who haven't configured it (keeping it working for existing installs). When it's done, a new user enters their credentials once, picks streams in one place, and never sees that there were ever two inputs.

We have a precedent to follow: the SentinelOne integration is doing the same consolidation (see #20286), and the `threat`-stream migration there is a usable template for the mechanics.

## Why it's worth doing

- Credentials entered once instead of twice. This is the headline fix.
- One input to explain in the docs and in support, instead of two.
- CEL-only improvements (state persistence, dedup, error handling) start covering every stream.

## What to watch out for

The upgrade is the risky part, not the build. Switching a stream's input type can reset its cursor and cause a gap or duplicate events at the cutover, and here it's 14 streams at once. We need a migration mechanism that preserves existing policies (input aliasing vs a `run_as_cel` shim vs rewritten CEL programs is still open), a plan for cursor continuity, and tests that prove no gaps or duplicates before release. Coordination with Fleet/Ecosystem is likely needed for the packaging and input-visibility changes. The changelog has to spell out what existing users should expect.

## Rough breakdown

1. Design: pick the migration mechanism and the existing-policy preservation strategy.
2. Migrate the Reports API HTTPJSON streams to CEL, in sensible batches.
3. Handle the stragglers (alert, gcp, and anything that doesn't fit the Reports API pattern) on their own.
4. Prove the upgrade is clean: no gaps, no duplicates at cutover.
5. Make CEL the default and hide HTTPJSON for new users; update docs and changelog.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.