opensearch-project / opensearch-project/data-prepper
[Idea] Log Generator Source
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
As a user of Data Prepper that wants to test Data Prepper log analytics with my logs rather quickly, it is a pain to have to configure an http or file source to send logs through Data Prepper.
As a developer of a new processor, it would be nice if I could test the end to end functionality of my processor without the http or file source.
Describe the solution you'd like
A source that generates either custom or common log formats that can be used to test pipelines, new processors, or to demo new processors.
Here is an example configuration which will choose a random log from a list of logs every 5 seconds and send it through Data Prepper until 20 logs have been generated.
source:
log_generator:
interval: 5
# This could default to create an infinite number of logs
count: 20
log_type:
custom:
# default for ordered will be false, which chooses a random log from log_lines
# ordered being true will cycle through the logs in the order they appear in `log_lines`
ordered: true
log_lines:
- 'This is my test log which will get the message key added since it is not json'
- '{"log": "This is a json string which will get converted to an Event"}'
- '{"key1": "value1", "key2": "value2"}'
In addition to custom logs, the log_generator could support pre made log types. Here is an example configuration where random logs in the apache common log format are generated. This idea can be expanded for many other types of common logs (syslog, s3, etc)
source:
log_generator:
log_type:
apache_clf:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue proposes a new log_generator source alongside the existing http and file sources, with YAML examples for custom and Apache common-format logs. Start by reviewing how those existing sources are configured and how generated events enter pipelines. Done means the proposed source can generate configured logs at an interval, optionally stop at a count, and support the described log types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100