elastic / elastic/integrations
[github] Consolidate the API data streams onto a single CEL input
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
# Consolidate the GitHub integration API streams onto a single CEL input
**Package:** `github` · **Type:** Enhancement · **Team:** Security Service Integrations
## The problem (user's view)
Collection is split across two Elastic Agent inputs, and the HTTPJSON side is the larger one. `security_advisories` runs on CEL, but `audit`, `code_scanning`, `dependabot`, `issues` and `secret_scanning` are all still on HTTPJSON. Because each input carries its own copy of the connection config, a user enters the same GitHub credentials twice and toggles streams in two places. It's just history — the newer stream got CEL, the rest never moved.
## Where we are
Six data streams (v2.25.0): `security_advisories` on CEL; `audit`, `code_scanning`, `dependabot`, `issues`, `secret_scanning` on HTTPJSON. This is a larger lift than the other consolidations because five streams remain.
## What I'm proposing
Migrate the five HTTPJSON streams to CEL so the integration runs through one input and the connection is configured once. Do it in sensible batches rather than all at once.
## Why it's worth doing
- Credentials entered once instead of twice.
- One engine to explain in docs and support.
- CEL-only improvements (state persistence, dedup, error handling) cover every stream.
## What to watch out for
The upgrade is the risky part, and it's five streams here. Switching input type can reset cursors and cause gaps or duplicates at cutover, so we need a cursor-continuity plan and tests proving no gaps/duplicates for each stream, plus a clear changelog note.
## Rough breakdown
1. Design: batch order and cursor-continuity strategy.
2. Migrate `audit`, `code_scanning`, `dependabot`, `issues`, `secret_scanning` to CEL in batches + tests.
3. Prove the upgrade is clean at cutover.
4. Update docs and changelog.
Part of a wider push to put Security Service Integrations packages on a single CEL input. Related: #20286 (SentinelOne), #14813 (Google Workspace), #20288 (Cortex XDR).
Contributor guide
Research direction
Start by comparing the existing `security_advisories` CEL stream with the HTTPJSON streams for `audit`, `code_scanning`, `dependabot`, `issues`, and `secret_scanning`. Define the batch order and cursor-continuity approach, then add tests showing no gaps or duplicates during cutover; update the integration documentation and changelog when the migration is complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github
- Domain
- api, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100