elastic / elastic/integrations

[Github] New members data stream for entity support

Open
#18,855 2 comments 0 reactions 1 assignee Claimed by @chemamartinez View on GitHub
enhancement Integration:github Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Summary

The GitHub integration currently collects security and operational data across six data
streams (`audit`, `issues`, `code_scanning`, `dependabot`, `secret_scanning`,
`security_advisories`), all of which are event or alert based. None of them collect
entity-state data; that is, a stable, current snapshot of who is a member of the
organization, what role they hold, which teams they belong to, and what their security
posture is.

This issue tracks the creation of a new `members` data stream that collects organization
member profiles and their associated attributes to support ECS `entity.*` field population
for GitHub user entities.

## Motivation

To support entity analytics workflows in Elastic (entity store, entity risk scoring, SIEM
identity graphs), the integration needs at least one data stream that represents entities
at rest rather than events over time. GitHub organization members are the primary identity
surface in GitHub and map directly to the ECS `user` entity type.

## Entity Type

The new data stream represents `user` entities, GitHub organization members (human
accounts and optionally bots). GitHub does not expose host or device entities via its API,
so `host.entity.*` fields are out of scope for this integration.

## Collection Strategy

The data stream will use a **GraphQL-first** approach, consistent with the existing
`dependabot` data stream which already uses `POST /graphql`. A single paginated GraphQL
query covers member identity, 2FA status, and team membership in one call. Supplementary
REST calls are used for membership role and optionally for owned repositories.

### Endpoints

| Call | Type | Purpose |
|------|------|---------|
| `POST https://api.github.com/graphql` / `organization.membersWithRole` | GraphQL | Primary collection: login, id, name, email, 2FA status, team memberships. Paginated via `pageInfo.endCursor`. |
| `GET /orgs/{org}/memberships/{username}` | REST | Per-member org role (`admin`, `member`, `billing_manager`) and effective permissions. |
| `GET /users/{username}/repos?type=owner` | REST | Repositories owned by the member. Opt-in O(n_members) calls. |

### Required OAuth Scopes (new)

| Scope | Reason |
|-------|--------|
| `read:org` | List members, read team membership, read org membership role |
| `read:user` | Read user profiles |
| `public_repo` or `repo` | Read owned repositories (only if the opt-in repos collection is enabled) |

Existing `read:audit_log` scope is unchanged.

## Manifest Variables (preliminary)

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `organization` | text | required | GitHub organization login name |
| `access_token` | password | required | PAT with `read:org` and `read:user` scopes |
| `interval` | text | `24h` | Polling interval |
| `collect_mfa_status` | bool | `false` | Enable 2FA status collection. Requires an org-owner token. |
| `collect_owned_repos` | bool | `false` | Enable per-member repository ownership collection. Adds O(n_members) REST calls per cycle. |
| `include_bots` | bool | `false` | Include bot accounts (type `Bot`) in collected members. |

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.