Graylog2 / Graylog2/graylog2-server

http JSON input -> JSON extractor, difficult to extract structured data values

Open
#2,751 9 comments 5 reactions 0 assignees View on GitHub
bug inputs P3 processing S3 triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

In reference to:

https://groups.google.com/d/msg/graylog2/js9dBgEHaDA/-jk09IHrAAAJ
## Expected Behavior

I've got a JSON HTTP API which returns several data items which are of interested to me, something like this:

``` json
{
"objs": { "devs": 100, "subs": 200 },
"dbstats": { "read": 300, "write": 400 }
}
```

I've defined an HTTP JSON _input_ in Graylog, and was hoping to then use the JSON extractor to map the values of `subs`, `devs`, `read`, `write` into properties of the collected messages (for monitoring, charts, etc.)
## Current Behavior

Let's focus on `subs` and `devs` for now.

The HTTP JSON input has its `JSON path of data to extract` set to `$.objs`.

And then I'm trying to use JSON extractor to break that apart into `devs` and `subs`.

But that does't work because the HTTP JSON input stores the data (in `result`) as some sort of Java map.toString(), not valid JSON, and then the JSON extractor can't work wit that:

Expected values in `result`:

`{ "devs": 100, "subs": 200 }`

Actual values in `result`:

`{ devs=100, subs=200 }`
## Possible Solution

1 - Change the HTTP JSON extractor so it can store its output (`result`) as JSON. Maybe a setting so it's optional, preserving the current behavior.

OR

2 - Add a way to configure JSON path -> value mapping right in the HTTP JSON extractor, allowing for multiple values.

So instead of the current (still not working, but maybe possible with solution #1):

JSON HTTP input -> calls the HTTP JSON API -> takes $.objs and stores as JSON -> JSON extractor takes this JSON apart and adds the values of `subs` and `devs` to each message --

-- it would be possible to do this:

JSON HTTP input -> calls the HTTP JSON API -> takes `$.objs.devs` and stores as `devs`, takes `$.objs.subs` and stores as `subs`, etc.

A JSON Extractor would not be needed then here.
## Context

Trying to collect structured typed values from an HTTP JSON API using a HTTP JSON input.
## Your Environment
- Graylog Version: 2.0.3
- Elasticsearch Version: 2.3.5
- MongoDB Version: 3.2.9
- Operating System: Debian Linux 8.5
- Browser version: Google Chrome 52

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.