influxdata / influxdata/telegraf

Parsing nested arrays containing multiple datasets in json_v2

Open
#11,467 2 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

Hello everybody.

I'm trying to parse mains grid frequency data from gridradar.net and got stuck. Not sure if it's me or a bug.
Every request returns mains frequency data since the last request, so the number of elements is not fixed.

If the data would look like this, everything would be easy:
```
[
{
"datapoints": [
{
"f":50.003,
"t":"2022-07-05 12:55:56"
},
{
"f":50.014,
"t":"2022-07-05 12:55:57"
},
{
"f":49.998,
"t":"2022-07-05 12:55:58"
},
{
"f":49.893,
"t":"2022-07-05 12:55:59"
}
],
"target": "median_frequency"
}
]
```

A corresponding telegraf.conf like this works well here:
```
[[inputs.file]]
files = ["./test3.json"]
data_format = "json_v2"
[[inputs.file.json_v2]]
measurement_name = "median_freq"
[[inputs.file.json_v2.object]]
path = "0.datapoints"
disable_prepend_keys = true
timestamp_format = "2006-01-02 15:04:05"
timestamp_key = "t"
```

Telegraf test output:
```
> median_freq, f=50.003 1657025756000000000
> median_freq, f=50.014 1657025757000000000
> median_freq, f=49.998 1657025758000000000
> median_freq, f=49.893 1657025759000000000
```

Now, the problem is that in reality the incoming data looks like this:
```
[
{
"datapoints": [
[
50.003,
"2022-07-05 12:55:56"
],
[
50.014,
"2022-07-05 12:55:57"
],
[
49.998,
"2022-07-05 12:55:58"
],
[
49.893,
"2022-07-05 12:55:59"
]
],
"target": "median_frequency"
}
]
```

Doing some gjson magic in GJSON Playground brought me to this conf:
```
[[inputs.file]]
files = ["./test2.json"]
data_format = "json_v2"
[[inputs.file.json_v2]]
measurement_name = "median_freq"
[[inputs.file.json_v2.object]]
path = "{datapoints:{f:0.datapoints.#.0,t:0.datapoints.#.1}.@group}"
disable_prepend_keys = true
timestamp_format = "2006-01-02 15:04:05"
timestamp_key = "datapoints.t"
```

Well, the test output is just empty.

Is this a bug, am I hitting some limitation of json_v2 or am I doing something wrong? On the Playground this looks like it should work.

Best,
Sebbi

Contributor guide

Open the contributing guide

Research direction

No source files or tests are identified in the report. Reproduce the two json_v2 configurations with the nested array payload and trace how the object path is evaluated; done means determining whether this shape is supported, then fixing the behavior or documenting the limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, json
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.