elastic / elastic/elastic-package

[system tests] Validate fields in transforms are documented based on mappings

Open
#2,341 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

Follows #2207

In order to try to be agnostic to the structure of the documents ingested to run the validation in system tests. It would be helpful to run validations comparing the mapping definitions instead.

For those mappings that can not be validated against the preview mappings in https://github.com/elastic/elastic-package/issues/2206, it is also needed to validate if they match with any of the dynamic templates found in the data stream.

As part of the packages, there could be defined transforms with their own fields that should also be validated.

Check and if possible validate those fields based on the available mappings and dynamic templates.

The destination index could be used in

Mappings and dynamic templates can be retrieved from these APIs:

- Mappings and dynamic templates installed by Fleet before ingesting any doc.
- These preview mappings can be retrieved using this simulated API:
```
POST /_index_template/_simulate/

# Example
POST /_index_template/_simulate/logs-microsoft_dhcp.log
```
- Mappings and dynamic templates that are present after ingesting the docs as part of the system tests.
- These mappings can be retrieved using this API:
```
GET //_mapping/
```

The destination index (`transforms.dest.index`) could be used in the above APIs:
```json
{
"count": 1,
"transforms": [
{
"id": "logs-ti_anomali.latest_intelligence-default-0.1.0",
"dest": {
"index": "logs-ti_anomali_latest.intelligence-1",
"aliases": [
{
"alias": "logs-ti_anomali_latest.intelligence",
"move_on_creation": true
}
]
},
```
The index template looks like it could also known in advance (suffix `-template`?)
```
GET /logs-ti_anomali_latest.intelligence-1/_mapping

# using index
POST /_index_template/_simulate_index/logs-ti_anomali_latest.intelligence-1

# using template
POST /_index_template/_simulate/logs-ti_anomali.latest_intelligence-template
```

To be tested:
- Run these validations in stack 7.x
- Run these validations in stack 8.x
- Run these validations in input and integration packages.
- Run these validations in Stacks with LogsDB enabled (synthetics).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the system-test validation flow and the code that defines or processes package transforms. Review the mapping and dynamic-template APIs described in the issue, including preview and post-ingestion retrieval. Done means transform fields are validated against available mappings or dynamic templates across 7.x, 8.x, input and integration packages, and LogsDB-enabled stacks.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
databases, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.