elastic / elastic/integrations

[Suricata]: X-Forwarded-For parsing

Open
#14,469 2 comments 0 reactions 0 assignees View on GitHub
Integration:suricata Stalled Team:Security-Deployment and Devices
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

Suricata [suricata]

### Dataset Name

_No response_

### Integration Version

2.25.0

### Agent Version

NA

### OS Version and Architecture

NA

### User Goal

The goal is for the Elastic Suricata integration to parse the values in HTTP X-Forwarded-For (xff) and put them into an ECS field.

I propose the extraction of the IPs from `http.xff` and placing them into ECS field [related.ip](https://www.elastic.co/docs/reference/ecs/ecs-related#field-related-ip). `related.ip` should contain an array of all IPs seen in the event.

### Existing Features

There does not appear to be a defined mapping for xff in the Elastic Suricata integration. The integration runs the JSON processor against `event.original` and the output written to [suricata.eve]( https://github.com/elastic/integrations/blob/main/packages/suricata/data_stream/eve/elasticsearch/ingest_pipeline/default.yml#L29) which is why the xff field ends up being created in `suricata.eve.xff` as a type keyword.

### What did you see?

## Sample Suricata Eve Output
Suricata will log the complete/original HTTP X-Forwarded-For (xff) header in the Eve output log in `http.xff`. If the Suricata additional xff parsing is [configured]( https://docs.suricata.io/en/latest/configuration/suricata-yaml.html#eve-extensible-event-format), and additional field `xff` is created with just a single IP.

Below are two examples of an Eve JSON output with the additional xff parsing enabled which is why there are two xff fields. Genereated with Suricata version 7.0.10.

```json
{
"timestamp": "2025-07-08T07:41:11.947702+0000",
"flow_id": 1,
"event_type": "http",
"src_ip": "192.168.1.100",
"src_port": 49518,
"dest_ip": "10.1.1.200",
"dest_port": 32400,
"proto": "TCP",
"pkt_src": "stream (flow timeout)",
"tx_id": 0,
"http": {
"http_port": 0,
"url": "/web/",
"http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
"xff": "192.168.1.100:1234",
"http_method": "GET",
"protocol": "HTTP/1.0",
"length": 0
},
"xff": "192.168.1.100"
}

{
"timestamp": "2025-07-08T07:41:11.947702+0000",
"flow_id": 2,
"event_type": "http",
"src_ip": "192.168.1.100",
"src_port": 49518,
"dest_ip": "10.1.1.200",
"dest_port": 32400,
"proto": "TCP",
"pkt_src": "stream (flow timeout)",
"tx_id": 0,
"http": {
"http_port": 0,
"url": "/web/",
"http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
"xff": "192.168.1.100:1234, 1.2.3.4:1234",
"http_method": "GET",
"protocol": "HTTP/1.0",
"length": 0
},
"xff": "1.2.3.4"
}
```

## Parsing Example

I came up with various X-Forwarded-For parsing scenerios partially based on the unit testing in [Suricata xff source code](https://github.com/OISF/suricata/blob/master/src/app-layer-htp-xff.c#L257) as well as [Mozilla reference documents]( https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For). Notice that sometimes there are multiple IPs, sometimes with ports, sometimes a mixture of IPv4 and IPv6.

Create a test index with test data
```
POST /_bulk
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"192.168.1.100"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"192.168.1.100:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[12::34]"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[12::34]:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"12::34"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[2a03:2880:1010:3f02:face:b00c:0:2]"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[2a03:2880:1010:3f02:face:b00c:0:2]:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[::ffff:1.2.3.4]:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"192.168.1.100:1234, 1.2.3.4:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"192.168.1.100:1234,1.2.3.4:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"1.2.3.4,+5.6.7.8,+9.10.11.12"}}
```

Here is a Pipeline to extract the IPs from `http.xff` and then put it into array `_tmp_.xff`. The could then be appended to `related.ip` (which isn't in the below).

```json
PUT _ingest/pipeline/suricata_xff
{
"processors": [
{
"gsub": {
"field": "http.xff",
"pattern": "(\\s|\\+)",
"replacement": ""
}
},
{
"split": {
"field": "http.xff",
"separator": ",",
"target_field": "_tmp_.xff",
"ignore_missing": true,
"ignore_failure": true
}
},
{
"gsub": {
"field": "_tmp_.xff",
"pattern": "\\]:[^,]+",
"replacement": "]",
"ignore_missing": true
}
},
{
"gsub": {
"field": "_tmp_.xff",
"pattern": "(\\[|\\])",
"replacement": "",
"ignore_missing": true,
"ignore_failure": true
}
},
{
"gsub": {
"field": "_tmp_.xff",
"pattern": "(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})(:[^\\s]+)",
"replacement": "$1"
}
}
]
}
```

Here is the output (reduced to highlight the parsing) of the processor applied to the data put into the suricata_xff test index.

```json
{"http":{"xff":"1.2.3.4,5.6.7.8,9.10.11.12"},"_tmp_":{"xff":["1.2.3.4","5.6.7.8","9.10.11.12"]}}
{"http":{"xff":"192.168.1.100:1234,1.2.3.4:1234"},"_tmp_":{"xff":["192.168.1.100","1.2.3.4"]}}
{"http":{"xff":"192.168.1.100:1234,1.2.3.4:1234"},"_tmp_":{"xff":["192.168.1.100","1.2.3.4"]}}
{"http":{"xff":"[::ffff:1.2.3.4]:1234"},"_tmp_":{"xff":["::ffff:1.2.3.4"]}}
{"http":{"xff":"[2a03:2880:1010:3f02:face:b00c:0:2]:1234"},"_tmp_":{"xff":["2a03:2880:1010:3f02:face:b00c:0:2"]}}
{"http":{"xff":"[2a03:2880:1010:3f02:face:b00c:0:2]"},"_tmp_":{"xff":["2a03:2880:1010:3f02:face:b00c:0:2"]}}
{"http":{"xff":"[12::34]:1234"},"_tmp_":{"xff":["12::34"]}}
{"http":{"xff":"192.168.1.100:1234"},"_tmp_":{"xff":["192.168.1.100"]}}
{"http":{"xff":"192.168.1.100"},"_tmp_":{"xff":["192.168.1.100"]}}
```

### Anything else?

### Possible issues:
The discussion in https://github.com/elastic/integrations/issues/7622 suggests there is not consensus to which ECS field an xff field should land.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.