elastic / elastic/integrations

[github] Consolidate the API data streams onto a single CEL input

Open
#20,293 2 comments 0 reactions 0 assignees View on GitHub
enhancement Integration:github Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

# Consolidate the GitHub integration API streams onto a single CEL input

**Package:** `github` · **Type:** Enhancement · **Team:** Security Service Integrations

## The problem (user's view)
Collection is split across two Elastic Agent inputs, and the HTTPJSON side is the larger one. `security_advisories` runs on CEL, but `audit`, `code_scanning`, `dependabot`, `issues` and `secret_scanning` are all still on HTTPJSON. Because each input carries its own copy of the connection config, a user enters the same GitHub credentials twice and toggles streams in two places. It's just history — the newer stream got CEL, the rest never moved.

## Where we are
Six data streams (v2.25.0): `security_advisories` on CEL; `audit`, `code_scanning`, `dependabot`, `issues`, `secret_scanning` on HTTPJSON. This is a larger lift than the other consolidations because five streams remain.

## What I'm proposing
Migrate the five HTTPJSON streams to CEL so the integration runs through one input and the connection is configured once. Do it in sensible batches rather than all at once.

## Why it's worth doing
- Credentials entered once instead of twice.
- One engine to explain in docs and support.
- CEL-only improvements (state persistence, dedup, error handling) cover every stream.

## What to watch out for
The upgrade is the risky part, and it's five streams here. Switching input type can reset cursors and cause gaps or duplicates at cutover, so we need a cursor-continuity plan and tests proving no gaps/duplicates for each stream, plus a clear changelog note.

## Rough breakdown
1. Design: batch order and cursor-continuity strategy.
2. Migrate `audit`, `code_scanning`, `dependabot`, `issues`, `secret_scanning` to CEL in batches + tests.
3. Prove the upgrade is clean at cutover.
4. Update docs and changelog.

Part of a wider push to put Security Service Integrations packages on a single CEL input. Related: #20286 (SentinelOne), #14813 (Google Workspace), #20288 (Cortex XDR).

Contributor guide

Open the contributing guide

Research direction

Start by comparing the existing `security_advisories` CEL stream with the HTTPJSON streams for `audit`, `code_scanning`, `dependabot`, `issues`, and `secret_scanning`. Define the batch order and cursor-continuity approach, then add tests showing no gaps or duplicates during cutover; update the integration documentation and changelog when the migration is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
github
Domain
api, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.