elastic / elastic/docs-content

[Internal]: Adding raw sample data to integration docs

Open
#8,250 1 comment 0 reactions 0 assignees View on GitHub
enhancement Feature: Security Integrations Team:Ingest triaged
Dominant language
No language data
Stars
47
Forks
261
Avg merge
3d 12h
Merged PRs (30d)
116

Description

### Description

Customers sending data through third party pipeline tools like Cribl have no way to know whether their data matches what our pipelines expect until they send it in production. When the shape differs, the pipeline can fail, the document still indexes with event.kind: pipeline_error, and detections and dashboards silently don't work.

Every data stream has pipeline test fixtures, e.g. [test-okta-system-events.log](https://github.com/elastic/integrations/blob/main/packages/okta/data_stream/system/_dev/test/pipeline/test-okta-system-events.log) - sample events from each datasource, already sanitised, already public and shows the format of the data we expect from each data source. However they live on Github and users are unaware they exist - they sit under _dev/test/pipeline/, where no customer will look.

Exposing these samples in the integration docs would let a customer diff their own data against ours before sending production traffic, and would make a pre-flight check possible — today there's no published expectation to validate against.

We currently expose 'Example' data for each integration after the data is processed by our pipeline. This ask is to also expose the raw samples prior to processing. As an example, here's the Okta raw samples in our Github repo: https://github.com/elastic/integrations/blob/main/packages/okta/data_stream/system/_dev/test/pipeline/test-okta-system-events.log

### Resources

n/a not tied to a specific feature

### Which deployment methods does this change impact?

Elastic On-Prem and Cloud (all)

### Feature differences

_No response_

### What Elastic Stack release is this request related to?

N/A

### Serverless release

_No response_

### Collaboration model

We expect to collaborate on the first draft

### Point of contact.

**Main contact:** @jamiehynds

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the existing raw fixture at packages/okta/data_stream/system/_dev/test/pipeline/test-okta-system-events.log and the integration documentation that currently exposes processed Example data. Add a customer-visible place for the sanitised raw samples, then verify that the raw and processed examples are both available for each applicable data stream.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.