elastic / elastic/integrations
[Suricata]: X-Forwarded-For parsing
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
Suricata [suricata]
### Dataset Name
_No response_
### Integration Version
2.25.0
### Agent Version
NA
### OS Version and Architecture
NA
### User Goal
The goal is for the Elastic Suricata integration to parse the values in HTTP X-Forwarded-For (xff) and put them into an ECS field.
I propose the extraction of the IPs from `http.xff` and placing them into ECS field [related.ip](https://www.elastic.co/docs/reference/ecs/ecs-related#field-related-ip). `related.ip` should contain an array of all IPs seen in the event.
### Existing Features
There does not appear to be a defined mapping for xff in the Elastic Suricata integration. The integration runs the JSON processor against `event.original` and the output written to [suricata.eve]( https://github.com/elastic/integrations/blob/main/packages/suricata/data_stream/eve/elasticsearch/ingest_pipeline/default.yml#L29) which is why the xff field ends up being created in `suricata.eve.xff` as a type keyword.
### What did you see?
## Sample Suricata Eve Output
Suricata will log the complete/original HTTP X-Forwarded-For (xff) header in the Eve output log in `http.xff`. If the Suricata additional xff parsing is [configured]( https://docs.suricata.io/en/latest/configuration/suricata-yaml.html#eve-extensible-event-format), and additional field `xff` is created with just a single IP.
Below are two examples of an Eve JSON output with the additional xff parsing enabled which is why there are two xff fields. Genereated with Suricata version 7.0.10.
```json
{
"timestamp": "2025-07-08T07:41:11.947702+0000",
"flow_id": 1,
"event_type": "http",
"src_ip": "192.168.1.100",
"src_port": 49518,
"dest_ip": "10.1.1.200",
"dest_port": 32400,
"proto": "TCP",
"pkt_src": "stream (flow timeout)",
"tx_id": 0,
"http": {
"http_port": 0,
"url": "/web/",
"http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
"xff": "192.168.1.100:1234",
"http_method": "GET",
"protocol": "HTTP/1.0",
"length": 0
},
"xff": "192.168.1.100"
}
{
"timestamp": "2025-07-08T07:41:11.947702+0000",
"flow_id": 2,
"event_type": "http",
"src_ip": "192.168.1.100",
"src_port": 49518,
"dest_ip": "10.1.1.200",
"dest_port": 32400,
"proto": "TCP",
"pkt_src": "stream (flow timeout)",
"tx_id": 0,
"http": {
"http_port": 0,
"url": "/web/",
"http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
"xff": "192.168.1.100:1234, 1.2.3.4:1234",
"http_method": "GET",
"protocol": "HTTP/1.0",
"length": 0
},
"xff": "1.2.3.4"
}
```
## Parsing Example
I came up with various X-Forwarded-For parsing scenerios partially based on the unit testing in [Suricata xff source code](https://github.com/OISF/suricata/blob/master/src/app-layer-htp-xff.c#L257) as well as [Mozilla reference documents]( https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For). Notice that sometimes there are multiple IPs, sometimes with ports, sometimes a mixture of IPv4 and IPv6.
Create a test index with test data
```
POST /_bulk
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"192.168.1.100"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"192.168.1.100:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[12::34]"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[12::34]:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"12::34"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[2a03:2880:1010:3f02:face:b00c:0:2]"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[2a03:2880:1010:3f02:face:b00c:0:2]:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"[::ffff:1.2.3.4]:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"192.168.1.100:1234, 1.2.3.4:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"192.168.1.100:1234,1.2.3.4:1234"}}
{ "index" : { "_index" : "suricatatest"} }
{"http":{"xff":"1.2.3.4,+5.6.7.8,+9.10.11.12"}}
```
Here is a Pipeline to extract the IPs from `http.xff` and then put it into array `_tmp_.xff`. The could then be appended to `related.ip` (which isn't in the below).
```json
PUT _ingest/pipeline/suricata_xff
{
"processors": [
{
"gsub": {
"field": "http.xff",
"pattern": "(\\s|\\+)",
"replacement": ""
}
},
{
"split": {
"field": "http.xff",
"separator": ",",
"target_field": "_tmp_.xff",
"ignore_missing": true,
"ignore_failure": true
}
},
{
"gsub": {
"field": "_tmp_.xff",
"pattern": "\\]:[^,]+",
"replacement": "]",
"ignore_missing": true
}
},
{
"gsub": {
"field": "_tmp_.xff",
"pattern": "(\\[|\\])",
"replacement": "",
"ignore_missing": true,
"ignore_failure": true
}
},
{
"gsub": {
"field": "_tmp_.xff",
"pattern": "(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})(:[^\\s]+)",
"replacement": "$1"
}
}
]
}
```
Here is the output (reduced to highlight the parsing) of the processor applied to the data put into the suricata_xff test index.
```json
{"http":{"xff":"1.2.3.4,5.6.7.8,9.10.11.12"},"_tmp_":{"xff":["1.2.3.4","5.6.7.8","9.10.11.12"]}}
{"http":{"xff":"192.168.1.100:1234,1.2.3.4:1234"},"_tmp_":{"xff":["192.168.1.100","1.2.3.4"]}}
{"http":{"xff":"192.168.1.100:1234,1.2.3.4:1234"},"_tmp_":{"xff":["192.168.1.100","1.2.3.4"]}}
{"http":{"xff":"[::ffff:1.2.3.4]:1234"},"_tmp_":{"xff":["::ffff:1.2.3.4"]}}
{"http":{"xff":"[2a03:2880:1010:3f02:face:b00c:0:2]:1234"},"_tmp_":{"xff":["2a03:2880:1010:3f02:face:b00c:0:2"]}}
{"http":{"xff":"[2a03:2880:1010:3f02:face:b00c:0:2]"},"_tmp_":{"xff":["2a03:2880:1010:3f02:face:b00c:0:2"]}}
{"http":{"xff":"[12::34]:1234"},"_tmp_":{"xff":["12::34"]}}
{"http":{"xff":"192.168.1.100:1234"},"_tmp_":{"xff":["192.168.1.100"]}}
{"http":{"xff":"192.168.1.100"},"_tmp_":{"xff":["192.168.1.100"]}}
```
### Anything else?
### Possible issues:
The discussion in https://github.com/elastic/integrations/issues/7622 suggests there is not consensus to which ECS field an xff field should land.
Contributor guide
Assessment
This issue has not been assessed yet.