elastic / elastic/docs-content
[Internal]: Adding raw sample data to integration docs
- Dominant language
- No language data
- Stars
- 47
- Forks
- 261
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 116
Description
### Description
Customers sending data through third party pipeline tools like Cribl have no way to know whether their data matches what our pipelines expect until they send it in production. When the shape differs, the pipeline can fail, the document still indexes with event.kind: pipeline_error, and detections and dashboards silently don't work.
Every data stream has pipeline test fixtures, e.g. [test-okta-system-events.log](https://github.com/elastic/integrations/blob/main/packages/okta/data_stream/system/_dev/test/pipeline/test-okta-system-events.log) - sample events from each datasource, already sanitised, already public and shows the format of the data we expect from each data source. However they live on Github and users are unaware they exist - they sit under _dev/test/pipeline/, where no customer will look.
Exposing these samples in the integration docs would let a customer diff their own data against ours before sending production traffic, and would make a pre-flight check possible — today there's no published expectation to validate against.
We currently expose 'Example' data for each integration after the data is processed by our pipeline. This ask is to also expose the raw samples prior to processing. As an example, here's the Okta raw samples in our Github repo: https://github.com/elastic/integrations/blob/main/packages/okta/data_stream/system/_dev/test/pipeline/test-okta-system-events.log
### Resources
n/a not tied to a specific feature
### Which deployment methods does this change impact?
Elastic On-Prem and Cloud (all)
### Feature differences
_No response_
### What Elastic Stack release is this request related to?
N/A
### Serverless release
_No response_
### Collaboration model
We expect to collaborate on the first draft
### Point of contact.
**Main contact:** @jamiehynds
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the existing raw fixture at packages/okta/data_stream/system/_dev/test/pipeline/test-okta-system-events.log and the integration documentation that currently exposes processed Example data. Add a customer-visible place for the sanitised raw samples, then verify that the raw and processed examples are both available for each applicable data stream.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100