elastic / elastic/logstash

Parse json by extract specify key.

Open
#16,379 0 comments 0 reactions 0 assignees View on GitHub
enhancement status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Hello there!
Our system generates a large amount of logs, each containing many unused fields. If we use the following json filter:
```
filter {
json {
source => "message"
target => "parsed"
remove_field => ["message"]
}
}
```
then all unused fields will be unmarshaled into key/value pairs, which leads to significant CPU cost.
Is there any method to achieve something like this:
```
json {
parse_keys => ["@timestamp", "message", "k8s_pod_namespace", "k8s_pod"]
source => "message"
target => "parsed"
remove_field => ["message"]
}
```
This way, only the specified keys will be parsed, potentially reducing CPU usage significantly. We believe this feature would be extremely beneficial for optimizing performance in environments with large log volumes.
Thank you for considering this request!

Contributor guide

Open the contributing guide

Research direction

The request concerns the Logstash json filter with source, target, remove_field, and proposed parse_keys settings. Start by locating that filter's implementation and parsing tests, then verify that selecting keys avoids processing unused fields while preserving the requested output and reducing CPU work.

Written by the indexing model from the issue text.

Assessment

Tech stack
json
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.