Graylog2 / Graylog2/graylog2-server
Beats input strips JSON arrays containing JSON objects from Packetbeat messages.
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
Packetbeat fields that contain a JSON array containing JSON objects have the contain JSON objects removed, resulting in an empty field.
## Expected Behavior
Packetbeat fields that contain a JSON array of JSON objects retain those values. The field should at least be stored as a string in order to allow parsing and processing with the JSON processing tools in the pipeline.
## Current Behavior
Packetbeat DNS logs contain a field `packetbeat_dns_answers` that is a list of the records that are captured when a DNS query+response are logged. This field is logged in Graylog as an empty list:

Configuring packetbeat to output to the console shows that the field does have a value though:
```
"dns": {
"additionals_count": 0,
"answers": [
{
"class": "IN",
"data": "2001:4998:124:1507::f001",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:44:3507::8000",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:44:3507::8001",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:24:120d::1:0",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:124:1507::f000",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:24:120d::1:1",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
}
],
```
I tested this by adding a field in the Packetbeat config:
```yaml
processors:
- add_fields:
target: _testjson
fields:
name: [{"testkey0": "value"},{"testkey1":"value1"}]
```
And the field was added but it was also added as an empty array:

## Possible Solution
## Steps to Reproduce (for bugs)
1. Download, configure, and run the latest Packetbeat agent and send logs to the Graylog beats input
2. Generate sample DNS traffic where the Packetbeat agent will see it
3. Examine the logs sent to Graylog, specifically the `packetbeat_dns_answers` field
## Context
The DNS answers is one of the more meaningful bits of information. It is almost impossible to reconstruct this information reliably from the remaining fields that are indexed.
## Your Environment
* Graylog Version: 5.2.4
* Java Version: openjdk 11.0.22 2024-01-16
* OpenSearch Version: 1.3.7
* MongoDB Version:
* Operating System: Ubuntu 22.04.4 LTS
* Browser version: Firefox 122.0.1
Contributor guide
Assessment
This issue has not been assessed yet.