opensearch-project / opensearch-project/data-prepper
[BUG] OSIS Filter Not Honored
@kkondaka is already working on this.
Since Nov 19, 2024.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Describe the bug
The AWS Managed Data Prepper (OSIS) Does not seem to consistently honor the filtering that I have configured.
To Reproduce
Steps to reproduce the behavior:
- Enable Security Lake with cloudtrail and eks audit logs
- Create Ingestion for Security Lake -> OpenSearch
- Update pipeline to include routes
route:
- eks-logs: '/metadata/product/name == "amazon_eks"'
- cloudtrail-logs: '/metadata/product/name == "cloudtrail"'
- Have the following sink configuration
sink:
- opensearch:
# Provide an AWS OpenSearch Service domain endpoint
hosts: [ <redacted> ]
routes: [eks-logs]
aws:
# Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com
sts_role_arn: "<redacted>"
# Provide the region of the domain.
region: "us-east-1"
# Enable the 'serverless' flag if the sink is an Amazon OpenSearch Serverless collection
serverless: false
index: "ocsf-${/metadata/version}-${/class_uid}-${/class_name}-eks-${/accountid}-%{yyyy.MM.dd}"
- opensearch:
# Provide an AWS OpenSearch Service domain endpoint
hosts: [ <redacted> ]
routes: [cloudtrail-logs]
aws:
# Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com
sts_role_arn: "<redacted>"
# Provide the region of the domain.
region: "us-east-1"
# Enable the 'serverless' flag if the sink is an Amazon OpenSearch Serverless collection
serverless: false
index: "ocsf-${/metadata/version}-${/class_uid}-${/class_name}-cloudtrail-${/accountid}-%{yyyy.MM.dd}"
Expected behavior
To create two indexes. One for 'eks' and one for 'cloudtrail'. Instead what happens is that it creates three indexes; one for eks, one for cloudtrail, and then a third that is unlabeled.
The unlabeled index logs contain 'metadata.product.name', and the product name is either 'cloudtrail' or 'amazon_eks'. This should have been routed to the appropriate route but is not.
Screenshots
Environment (please complete the following information):
- AWS Managed OSIS
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.