elastic / elastic/integrations
[Netskope] Add support for pulling data directly from the Netskope REST API
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Summary
The current Netskope integration can only receive data that Netskope pushes out - via the TCP input (Cloud Log Shipper) or via cloud storage Log Streaming (AWS S3/SQS, Azure Blob, GCS). It has no way to pull data directly from the Netskope REST API.
This means customers who cannot (or do not want to) stand up Cloud Log Shipper / Cloud Exchange or LogStreaming to S3 have no supported path to get Netskope data into Elastic. Many competing SIEM/observability tools already offer a direct API-pull collector for Netskope.
This issue proposes adding a REST API polling collection method to the Netskope integration using Netskope's REST API v2 `dataexport` iterator endpoints.
## Why (customer/business value)
- Removes the dependency on Cloud Log Shipper / Cloud Exchange.
- Simplest possible onboarding: a customer generates a v2 API token and points the integration at their tenant URL — no forwarding infrastructure.
- Brings the Netskope integration to parity with how other vendors' collectors ingest Netskope data.
- Enables collection of data types that are awkward to get via streaming (e.g. audit, incident, client status).
## Netskope REST API v2 — technical details
**Auth:** v2 API token passed in the `Netskope-Api-Token` request header. Token is generated in the Netskope UI (Settings > Administration > Administrators & Roles / API Tokens). Read-only scope on the relevant `/api/v2/events/dataexport/*` endpoints is sufficient.
**Base URL:** `https://.goskope.com/api/v2/events/dataexport/...`
**We want to support both Events and Alerts**, across all of their respective types, via the v2 `dataexport` iterator endpoints:
**Events** — `/api/v2/events/dataexport/events/{type}`:
- `application`
- `page`
- `network`
- `audit`
- `connection`
- `incident`
- `endpoint`
- `infrastructure`
- `alert`
**Alerts** — `/api/v2/events/dataexport/alerts/{type}`:
- `uba`
- `securityassessment`
- `quarantine`
- `remediation`
- `policy`
- `malware`
- `malsite`
- `compromisedcredential`
- `ctep`
- `dlp`
- `watchlist`
Docs:
- REST API v2 dataexport iterator endpoints: https://docs.netskope.com/en/using-the-rest-api-v2-dataexport-iterator-endpoints/
- REST API v2 overview: https://docs.netskope.com/en/rest-api-v2-overview-312207
- API tokens: https://docs.netskope.com/en/api-tokens-2
## Proposed implementation
**⚠️ Do not assume the payload format is identical to the streamed/shipped data.** The three delivery methods differ:
- **REST API v2 dataexport** returns a paginated JSON *envelope* (a `result` array of event objects plus iterator metadata such as `wallclock`, `ok`, `status`, and the watermark/index) that must be unwrapped.
- **Log Streaming** delivers gzipped JSON files (one event per line) wrapped in a storage-object envelope.
- **Cloud Log Shipper (CLS via Cloud Exchange)** can transform/subset fields via the CLS Mapping wizard, so it is not a raw passthrough.
The inner event/alert objects are largely the same underlying Netskope schema, so the existing `netskope.*` / ECS field mappings should be *mostly* reusable — but the API path will at minimum need its own preprocessing stage to unwrap the `result` array and drop iterator metadata. **Each event and alert type must be validated against real API sample payloads** to confirm field-level parity before reusing a pipeline; expect some per-type divergence.
Configuration surface:
- Tenant hostname/URL
- v2 API token (secret)
- Selectable list of event types and alert types to collect
- Initial lookback (start epoch)
- Polling interval
Contributor guide
Assessment
This issue has not been assessed yet.