opensearch-project / opensearch-project/data-prepper

[Idea] Log Generator Source

Open
#980 2 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

backlog ease-of-use plugin - source
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
As a user of Data Prepper that wants to test Data Prepper log analytics with my logs rather quickly, it is a pain to have to configure an http or file source to send logs through Data Prepper.

As a developer of a new processor, it would be nice if I could test the end to end functionality of my processor without the http or file source.

Describe the solution you'd like
A source that generates either custom or common log formats that can be used to test pipelines, new processors, or to demo new processors.

Here is an example configuration which will choose a random log from a list of logs every 5 seconds and send it through Data Prepper until 20 logs have been generated.

source:
  log_generator:
    interval: 5
    # This could default to create an infinite number of logs
    count: 20 
    log_type:
      custom:
        #  default for ordered will be false, which chooses a random log from log_lines
        # ordered being true will cycle through the logs in the order they appear in `log_lines`      
        ordered: true
        log_lines: 
           - 'This is my test log which will get the message key added since it is not json'
           - '{"log": "This is a json string which will get converted to an Event"}'
           - '{"key1": "value1", "key2": "value2"}'              

In addition to custom logs, the log_generator could support pre made log types. Here is an example configuration where random logs in the apache common log format are generated. This idea can be expanded for many other types of common logs (syslog, s3, etc)

source:
  log_generator:
     log_type:
       apache_clf:     

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue proposes a new log_generator source alongside the existing http and file sources, with YAML examples for custom and Apache common-format logs. Start by reviewing how those existing sources are configured and how generated events enter pipelines. Done means the proposed source can generate configured logs at an interval, optionally stop at a count, and support the described log types.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.