elastic / elastic/elastic-package

elastic-package test pipeline fails when reroute processors are defined in both pipeline and routing_rules.yml

Open
#1,387 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

When routing_rules.yml looks like this:
```
- source_dataset: awsfirehose.log
rules:
- target_dataset: aws.cloudtrail
if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
namespace:
- "{{labels.data_stream.namespace}}"
- default
```

and ingest_pipeline/default.yml looks like this:
```
---
description: Pipeline for rerouting logs streams from Amazon Kinesis Data Firehose.
processors:
- set:
field: ecs.version
value: 8.0.0
- set:
field: cloud.provider
value: aws
- reroute:
if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
dataset: aws.cloudtrail
namespace: default
on_failure:
- set:
field: error.message
value: "{{ _ingest.on_failure_message }}"
```

I get this error when I run `elastic-package test pipeline -d log --generate`:
```
kaiyansheng ~/go/src/github.com/elastic/integrations/packages/awsfirehose [firehose_integration_package] $ elastic-package test pipeline -d log --generate
2023/08/09 16:25:45 WARN CommitHash is undefined, in both /Users/kaiyansheng/.elastic-package/version and the compiled binary, config may be out of date.
Run pipeline tests for the package
Error: error running package pipeline tests: could not complete test run: unmarshalling ingest pipeline content failed: yaml: unmarshal errors:
line 14: cannot unmarshal !!str `aws.clo...` into []string
line 16: cannot unmarshal !!str `default` into []string
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running `elastic-package test pipeline -d log --generate` against the shown `routing_rules.yml` and `ingest_pipeline/default.yml` inputs. Trace how those files are unmarshalled during pipeline testing, then verify that packages defining reroute processors in both locations complete the generated pipeline test without the reported YAML type errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.