elastic / elastic/elastic-package
[Meta] Validation of transforms defined in packages
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
Relates https://github.com/elastic/elastic-package/issues/1402
Relates https://github.com/elastic/elastic-package/issues/2341
Relates https://github.com/elastic/elastic-package/issues/2676
Currently, `elastic-package` performs a validation process for packages containing transform definitions based on validating the documents returned by the [transform preview API](https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-transform.html). This process was introduced in https://github.com/elastic/elastic-package/issues/1402.
As discovered in https://github.com/elastic/elastic-package/issues/2341, validating transforms in a proper way could be problematic as it is now. Main issues found:
- Some packages define `logs-*` as the source index (see https://github.com/elastic/elastic-package/issues/2341#issuecomment-2648457676).
- Difficult to control what documents are going to be ingested in the destination index.
- Is this a valid usage in an `integration` package?
- Does it make sense for a `content` package?
- Currently, it is not supported defining transforms in `content` packages ([spec link](https://github.com/elastic/package-spec/tree/0b0fb936fb576bbd5da41677c7460ee8f4b69cb9/spec/content)).
- Some fields are defined as `constant_keyword` in transforms (e.g. data_stream.dataset).
- If those fields come from different source data streams, there could be issues processing documents in the transform since those fields could have different values in different documents.
- Example error:
> Preview of field's value: '59467'; java.lang.IllegalArgumentException: [constant_keyword] field [data_stream.namespace] only accepts values that are equal to the value defined in the mappings [75519], but got [59467]]
- If validation takes into account the documents processed by the transforms (as it was tested with validation based on mappings):
- Testing times are increased considerably, as it can be checked in https://github.com/elastic/elastic-package/pull/2347
- Details in: https://github.com/elastic/elastic-package/issues/2341#issuecomment-2636828163
- Packages containing several system test configuration files (test cases) for the same data stream could not detect all errors/failures:
- It's likely that the second test case (configuration file) does not wait for the expected documents, since the transform already processed documents for the first test case.
- More information about this scenario in https://github.com/elastic/elastic-package/issues/2341#issuecomment-2621974145
Due to the above, it looks like that the testing and validation process for transforms should be revisited. Maybe they should have their own test suite along with pipeline, system, static, and other test suites.
cc @elastic/ecosystem
Related PRs:
- https://github.com/elastic/elastic-package/pull/2242
- (Attempt to add validation based on mappings): https://github.com/elastic/elastic-package/pull/2347
~Other issues related (should it be moved as sub-issue here?):~
- ~https://github.com/elastic/elastic-package/issues/2341~
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the current transform validation path and the existing pipeline, system, static, and other test suites. Review related issues 1402, 2341, and 2676, the transform preview API, and PRs 2242 and 2347; done means an agreed validation and testing approach addressing the listed source-index, constant_keyword, timing, and multiple-test-case concerns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- cli, testing, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100