opensearch-project / opensearch-project/data-prepper

Add WAF log convertor before sending to OpenSearch

Open
#2,305 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
It would be nice to have a convertor for WAF logs before sending to OpenSearch. If we want to build some Dashboard.

Describe the solution you'd like

  • We want Data-prepper parse the web ACL name before sending to OpenSearch
  • We need to distinguish between host and userAgent from ["httpRequest"]["headers"]

Additional context
Here are some parser code in Solution Centralized Logging with OpenSearch

class WAF(LogType):
    """An implementation of LogType for WAF Logs"""

    _format = "json"

    def parse(self, line: str):
        try:
            json_record = json.loads(line)

            # Extract web acl name, host and user agent
            json_record["webaclName"] = re.search(
                "[^/]/webacl/([^/]*)", json_record["webaclId"]
            ).group(1)
            headers = json_record["httpRequest"]["headers"]
            for header in headers:
                if header["name"].lower() == "host":
                    json_record["host"] = header["value"]
                elif header["name"].lower() == "user-agent":
                    json_record["userAgent"] = header["value"]
                else:
                    continue
            return json_record
        except Exception as e:
            logger.error(e)
            return {}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file or test is named; start by locating Data Prepper's WAF log conversion path and the code that sends records to OpenSearch. Use the supplied parser as the behavioral reference: done means extracting the web ACL name and distinguishing host from userAgent in httpRequest headers before forwarding the record.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, data, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.