elastic / elastic/integrations

[Netskope] Add support for pulling data directly from the Netskope REST API

Open
#20,885 1 comment 0 reactions 1 assignee Claimed by @moxarth-rathod View on GitHub
enhancement Integration:netskope Team:SDE-Crest Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Summary
The current Netskope integration can only receive data that Netskope pushes out - via the TCP input (Cloud Log Shipper) or via cloud storage Log Streaming (AWS S3/SQS, Azure Blob, GCS). It has no way to pull data directly from the Netskope REST API.

This means customers who cannot (or do not want to) stand up Cloud Log Shipper / Cloud Exchange or LogStreaming to S3 have no supported path to get Netskope data into Elastic. Many competing SIEM/observability tools already offer a direct API-pull collector for Netskope.

This issue proposes adding a REST API polling collection method to the Netskope integration using Netskope's REST API v2 `dataexport` iterator endpoints.

## Why (customer/business value)
- Removes the dependency on Cloud Log Shipper / Cloud Exchange.
- Simplest possible onboarding: a customer generates a v2 API token and points the integration at their tenant URL — no forwarding infrastructure.
- Brings the Netskope integration to parity with how other vendors' collectors ingest Netskope data.
- Enables collection of data types that are awkward to get via streaming (e.g. audit, incident, client status).

## Netskope REST API v2 — technical details

**Auth:** v2 API token passed in the `Netskope-Api-Token` request header. Token is generated in the Netskope UI (Settings > Administration > Administrators & Roles / API Tokens). Read-only scope on the relevant `/api/v2/events/dataexport/*` endpoints is sufficient.

**Base URL:** `https://.goskope.com/api/v2/events/dataexport/...`

**We want to support both Events and Alerts**, across all of their respective types, via the v2 `dataexport` iterator endpoints:

**Events** — `/api/v2/events/dataexport/events/{type}`:
- `application`
- `page`
- `network`
- `audit`
- `connection`
- `incident`
- `endpoint`
- `infrastructure`
- `alert`

**Alerts** — `/api/v2/events/dataexport/alerts/{type}`:
- `uba`
- `securityassessment`
- `quarantine`
- `remediation`
- `policy`
- `malware`
- `malsite`
- `compromisedcredential`
- `ctep`
- `dlp`
- `watchlist`

Docs:
- REST API v2 dataexport iterator endpoints: https://docs.netskope.com/en/using-the-rest-api-v2-dataexport-iterator-endpoints/
- REST API v2 overview: https://docs.netskope.com/en/rest-api-v2-overview-312207
- API tokens: https://docs.netskope.com/en/api-tokens-2

## Proposed implementation

**⚠️ Do not assume the payload format is identical to the streamed/shipped data.** The three delivery methods differ:
- **REST API v2 dataexport** returns a paginated JSON *envelope* (a `result` array of event objects plus iterator metadata such as `wallclock`, `ok`, `status`, and the watermark/index) that must be unwrapped.
- **Log Streaming** delivers gzipped JSON files (one event per line) wrapped in a storage-object envelope.
- **Cloud Log Shipper (CLS via Cloud Exchange)** can transform/subset fields via the CLS Mapping wizard, so it is not a raw passthrough.

The inner event/alert objects are largely the same underlying Netskope schema, so the existing `netskope.*` / ECS field mappings should be *mostly* reusable — but the API path will at minimum need its own preprocessing stage to unwrap the `result` array and drop iterator metadata. **Each event and alert type must be validated against real API sample payloads** to confirm field-level parity before reusing a pipeline; expect some per-type divergence.

Configuration surface:
- Tenant hostname/URL
- v2 API token (secret)
- Selectable list of event types and alert types to collect
- Initial lookback (start epoch)
- Polling interval

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.