Graylog2 / Graylog2/graylog2-server

Beats input strips JSON arrays containing JSON objects from Packetbeat messages.

Open
#18,416 13 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

Packetbeat fields that contain a JSON array containing JSON objects have the contain JSON objects removed, resulting in an empty field.

## Expected Behavior

Packetbeat fields that contain a JSON array of JSON objects retain those values. The field should at least be stored as a string in order to allow parsing and processing with the JSON processing tools in the pipeline.

## Current Behavior

Packetbeat DNS logs contain a field `packetbeat_dns_answers` that is a list of the records that are captured when a DNS query+response are logged. This field is logged in Graylog as an empty list:

![image](https://github.com/Graylog2/graylog2-server/assets/61566003/5570e041-2a10-4cae-9390-e96e2beb2b58)

Configuring packetbeat to output to the console shows that the field does have a value though:

```
"dns": {
"additionals_count": 0,
"answers": [
{
"class": "IN",
"data": "2001:4998:124:1507::f001",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:44:3507::8000",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:44:3507::8001",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:24:120d::1:0",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:124:1507::f000",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
},
{
"class": "IN",
"data": "2001:4998:24:120d::1:1",
"name": "yahoo.com",
"ttl": "114",
"type": "AAAA"
}
],
```

I tested this by adding a field in the Packetbeat config:
```yaml
processors:
- add_fields:
target: _testjson
fields:
name: [{"testkey0": "value"},{"testkey1":"value1"}]
```

And the field was added but it was also added as an empty array:

![image](https://github.com/Graylog2/graylog2-server/assets/61566003/a330b7c7-2fdd-4157-87f9-ae25ffbc2783)

## Possible Solution

## Steps to Reproduce (for bugs)

1. Download, configure, and run the latest Packetbeat agent and send logs to the Graylog beats input
2. Generate sample DNS traffic where the Packetbeat agent will see it
3. Examine the logs sent to Graylog, specifically the `packetbeat_dns_answers` field

## Context

The DNS answers is one of the more meaningful bits of information. It is almost impossible to reconstruct this information reliably from the remaining fields that are indexed.

## Your Environment

* Graylog Version: 5.2.4
* Java Version: openjdk 11.0.22 2024-01-16
* OpenSearch Version: 1.3.7
* MongoDB Version:
* Operating System: Ubuntu 22.04.4 LTS
* Browser version: Firefox 122.0.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.