influxdata / influxdata/telegraf
Parsing nested arrays containing multiple datasets in json_v2
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
Hello everybody.
I'm trying to parse mains grid frequency data from gridradar.net and got stuck. Not sure if it's me or a bug.
Every request returns mains frequency data since the last request, so the number of elements is not fixed.
If the data would look like this, everything would be easy:
```
[
{
"datapoints": [
{
"f":50.003,
"t":"2022-07-05 12:55:56"
},
{
"f":50.014,
"t":"2022-07-05 12:55:57"
},
{
"f":49.998,
"t":"2022-07-05 12:55:58"
},
{
"f":49.893,
"t":"2022-07-05 12:55:59"
}
],
"target": "median_frequency"
}
]
```
A corresponding telegraf.conf like this works well here:
```
[[inputs.file]]
files = ["./test3.json"]
data_format = "json_v2"
[[inputs.file.json_v2]]
measurement_name = "median_freq"
[[inputs.file.json_v2.object]]
path = "0.datapoints"
disable_prepend_keys = true
timestamp_format = "2006-01-02 15:04:05"
timestamp_key = "t"
```
Telegraf test output:
```
> median_freq, f=50.003 1657025756000000000
> median_freq, f=50.014 1657025757000000000
> median_freq, f=49.998 1657025758000000000
> median_freq, f=49.893 1657025759000000000
```
Now, the problem is that in reality the incoming data looks like this:
```
[
{
"datapoints": [
[
50.003,
"2022-07-05 12:55:56"
],
[
50.014,
"2022-07-05 12:55:57"
],
[
49.998,
"2022-07-05 12:55:58"
],
[
49.893,
"2022-07-05 12:55:59"
]
],
"target": "median_frequency"
}
]
```
Doing some gjson magic in GJSON Playground brought me to this conf:
```
[[inputs.file]]
files = ["./test2.json"]
data_format = "json_v2"
[[inputs.file.json_v2]]
measurement_name = "median_freq"
[[inputs.file.json_v2.object]]
path = "{datapoints:{f:0.datapoints.#.0,t:0.datapoints.#.1}.@group}"
disable_prepend_keys = true
timestamp_format = "2006-01-02 15:04:05"
timestamp_key = "datapoints.t"
```
Well, the test output is just empty.
Is this a bug, am I hitting some limitation of json_v2 or am I doing something wrong? On the Playground this looks like it should work.
Best,
Sebbi
Contributor guide
Research direction
No source files or tests are identified in the report. Reproduce the two json_v2 configurations with the nested array payload and trace how the object path is evaluated; done means determining whether this shape is supported, then fixing the behavior or documenting the limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, json
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100