Parse json by extract specify key.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
Hello there!
Our system generates a large amount of logs, each containing many unused fields. If we use the following json filter:
```
filter {
json {
source => "message"
target => "parsed"
remove_field => ["message"]
}
}
```
then all unused fields will be unmarshaled into key/value pairs, which leads to significant CPU cost.
Is there any method to achieve something like this:
```
json {
parse_keys => ["@timestamp", "message", "k8s_pod_namespace", "k8s_pod"]
source => "message"
target => "parsed"
remove_field => ["message"]
}
```
This way, only the specified keys will be parsed, potentially reducing CPU usage significantly. We believe this feature would be extremely beneficial for optimizing performance in environments with large log volumes.
Thank you for considering this request!
Contributor guide
Research direction
The request concerns the Logstash json filter with source, target, remove_field, and proposed parse_keys settings. Start by locating that filter's implementation and parsing tests, then verify that selecting keys avoids processing unused fields while preserving the requested output and reducing CPU work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100