elastic / elastic/elastic-package

[Discuss] Store a encoded copy of the original document for fields validation

Open
#2,016 1 comment 1 reaction 1 assignee Claimed by @mrodm View on GitHub
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

Summary
----

Store a encoded copy of the original document with a processor in the final pipeline, before ingestion, and use this copy to validate fields and generate sample documents instead of rebuilding the document from the ingested data.

Split fields validations in two sets, one that uses this encoded copy, and another one for the indexed data.

Background
----

When validating fields we use the documents as they are stored in Elasticsearch. With the adoption of features like `constant_keyword`, runtime fields, `synthetic` index mode, or `index: false` in packages it can be difficult to rebuild the original document. Some mappings could also introduce additional multifields, that in some cases we are ignoring, or have to ignore.

We have now quite some code attempting to handle all these cases, and corner cases in combinations between them. Every time a new feature of this kind is added new corner cases appear.

Going back to the original objectives of these tests, we want to validate these two things:
1. That the package is generating the expected data.
2. That all the fields available to users are documented.

With the current approach of checking the documents ingested as returned by the search API, we are missing the first point, as in many cases we don't have the data the package is generating, and we attempt to rebuild the documents from the indexed data.

So the proposal would be to explicitly split validations in two:
1. Validations on the data generated by the package. They should be based on the resulting data after applying all the pipelines, but before ingesting. For that we need some way to store this data, this could be done by storing a encoded copy of the document in the document itself with a processor in the `final_pipeline`.
2. Validations on the ingested data, similar to the ones we have now, but they could be relaxed to validate only that the fields are documented, and ignore their values.

Some tests will do only one set of validations or both. The encoded copy could be additionally used for the generation of sample documents.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.