elastic / elastic/elastic-package
elastic-package test pipeline fails when reroute processors are defined in both pipeline and routing_rules.yml
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
When routing_rules.yml looks like this:
```
- source_dataset: awsfirehose.log
rules:
- target_dataset: aws.cloudtrail
if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
namespace:
- "{{labels.data_stream.namespace}}"
- default
```
and ingest_pipeline/default.yml looks like this:
```
---
description: Pipeline for rerouting logs streams from Amazon Kinesis Data Firehose.
processors:
- set:
field: ecs.version
value: 8.0.0
- set:
field: cloud.provider
value: aws
- reroute:
if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
dataset: aws.cloudtrail
namespace: default
on_failure:
- set:
field: error.message
value: "{{ _ingest.on_failure_message }}"
```
I get this error when I run `elastic-package test pipeline -d log --generate`:
```
kaiyansheng ~/go/src/github.com/elastic/integrations/packages/awsfirehose [firehose_integration_package] $ elastic-package test pipeline -d log --generate
2023/08/09 16:25:45 WARN CommitHash is undefined, in both /Users/kaiyansheng/.elastic-package/version and the compiled binary, config may be out of date.
Run pipeline tests for the package
Error: error running package pipeline tests: could not complete test run: unmarshalling ingest pipeline content failed: yaml: unmarshal errors:
line 14: cannot unmarshal !!str `aws.clo...` into []string
line 16: cannot unmarshal !!str `default` into []string
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running `elastic-package test pipeline -d log --generate` against the shown `routing_rules.yml` and `ingest_pipeline/default.yml` inputs. Trace how those files are unmarshalled during pipeline testing, then verify that packages defining reroute processors in both locations complete the generated pipeline test without the reported YAML type errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100