opensearch-project / opensearch-project/data-prepper

[BUG] OSIS Filter Not Honored

Open
#5,200 0 comments 0 reactions 1 assignee View on GitHub

@kkondaka is already working on this.

Since Nov 19, 2024.

bug
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Describe the bug
The AWS Managed Data Prepper (OSIS) Does not seem to consistently honor the filtering that I have configured.

To Reproduce
Steps to reproduce the behavior:

  1. Enable Security Lake with cloudtrail and eks audit logs
  2. Create Ingestion for Security Lake -> OpenSearch
  3. Update pipeline to include routes
  route:
    - eks-logs: '/metadata/product/name == "amazon_eks"'
    - cloudtrail-logs: '/metadata/product/name == "cloudtrail"'
  1. Have the following sink configuration
sink:
    - opensearch:
        # Provide an AWS OpenSearch Service domain endpoint
        hosts: [ <redacted> ]
        routes: [eks-logs]
        aws:
          # Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com
          sts_role_arn: "<redacted>"
          # Provide the region of the domain.
          region: "us-east-1"
          # Enable the 'serverless' flag if the sink is an Amazon OpenSearch Serverless collection
          serverless: false
        index: "ocsf-${/metadata/version}-${/class_uid}-${/class_name}-eks-${/accountid}-%{yyyy.MM.dd}"
    - opensearch:
        # Provide an AWS OpenSearch Service domain endpoint
        hosts: [ <redacted> ]
        routes: [cloudtrail-logs]
        aws:
          # Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com
          sts_role_arn: "<redacted>"
          # Provide the region of the domain.
          region: "us-east-1"
          # Enable the 'serverless' flag if the sink is an Amazon OpenSearch Serverless collection
          serverless: false
        index: "ocsf-${/metadata/version}-${/class_uid}-${/class_name}-cloudtrail-${/accountid}-%{yyyy.MM.dd}"

Expected behavior
To create two indexes. One for 'eks' and one for 'cloudtrail'. Instead what happens is that it creates three indexes; one for eks, one for cloudtrail, and then a third that is unlabeled.

The unlabeled index logs contain 'metadata.product.name', and the product name is either 'cloudtrail' or 'amazon_eks'. This should have been routed to the appropriate route but is not.

Screenshots

Screenshot 2024-11-18 at 2 24 04 PM

Environment (please complete the following information):

  • AWS Managed OSIS

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.