elastic / elastic/elastic-package
[system tests] Validate fields in transforms are documented based on mappings
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
Follows #2207
In order to try to be agnostic to the structure of the documents ingested to run the validation in system tests. It would be helpful to run validations comparing the mapping definitions instead.
For those mappings that can not be validated against the preview mappings in https://github.com/elastic/elastic-package/issues/2206, it is also needed to validate if they match with any of the dynamic templates found in the data stream.
As part of the packages, there could be defined transforms with their own fields that should also be validated.
Check and if possible validate those fields based on the available mappings and dynamic templates.
The destination index could be used in
Mappings and dynamic templates can be retrieved from these APIs:
- Mappings and dynamic templates installed by Fleet before ingesting any doc.
- These preview mappings can be retrieved using this simulated API:
```
POST /_index_template/_simulate/
# Example
POST /_index_template/_simulate/logs-microsoft_dhcp.log
```
- Mappings and dynamic templates that are present after ingesting the docs as part of the system tests.
- These mappings can be retrieved using this API:
```
GET //_mapping/
```
The destination index (`transforms.dest.index`) could be used in the above APIs:
```json
{
"count": 1,
"transforms": [
{
"id": "logs-ti_anomali.latest_intelligence-default-0.1.0",
"dest": {
"index": "logs-ti_anomali_latest.intelligence-1",
"aliases": [
{
"alias": "logs-ti_anomali_latest.intelligence",
"move_on_creation": true
}
]
},
```
The index template looks like it could also known in advance (suffix `-template`?)
```
GET /logs-ti_anomali_latest.intelligence-1/_mapping
# using index
POST /_index_template/_simulate_index/logs-ti_anomali_latest.intelligence-1
# using template
POST /_index_template/_simulate/logs-ti_anomali.latest_intelligence-template
```
To be tested:
- Run these validations in stack 7.x
- Run these validations in stack 8.x
- Run these validations in input and integration packages.
- Run these validations in Stacks with LogsDB enabled (synthetics).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the system-test validation flow and the code that defines or processes package transforms. Review the mapping and dynamic-template APIs described in the issue, including preview and post-ingestion retrieval. Done means transform fields are validated against available mappings or dynamic templates across 7.x, 8.x, input and integration packages, and LogsDB-enabled stacks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- databases, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100