elastic / elastic/integrations
[pfsense] Parsing errors when rule action is Match
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
pfSense [pfsense]
### Dataset Name
pfsense.log
### Integration Version
1.19.1
### Agent Version
8.14.3
### Agent Output Type
elasticsearch
### Elasticsearch Version
8.14.3
### OS Version and Architecture
Security Onion 2.4.100 (Oracle Linux Server 9.5)
### Software/API Version
_No response_
### Error Message
_No response_
### Event Original
_No response_
### What did you do?
logs-pfsense.log-1.19.1-firewall (fixed):
[
{
"grok": {
"field": "message",
"patterns": [
"%{PF_LOG_ENTRY}%{GREEDYDATA}"
],
"pattern_definitions": {
"PF_LOG_ENTRY": "%{PF_LOG_DATA}%{PF_IP_SPECIFIC_DATA}%{PF_IP_DATA}%{PF_PROTOCOL_DATA}?",
"PF_LOG_DATA": "%{INT},%{INT}?,,%{WORD:rule.id},%{DATA:observer.ingress.interface.name},%{PF_REASON:event.reason},(?[^,]+),%{WORD:network.direction},",
"PF_REASON": "[a-zA-Z-]+",
"PF_IP_DATA": "%{NONNEGINT:network.bytes:long},%{IP:source.address},%{IP:destination.address},",
"PF_IP_SPECIFIC_DATA": "%{PF_IPv4_SPECIFIC_DATA}|%{PF_IPv6_SPECIFIC_DATA}",
"PF_IPv4_SPECIFIC_DATA": "(?(4)),%{BASE16NUM:pfsense.ip.tos},%{WORD:pfsense.ip.ecn}?,%{NONNEGINT:pfsense.ip.ttl:long},%{NONNEGINT:pfsense.ip.id:long},%{NONNEGINT:pfsense.ip.offset:long},(?:%{WORD:pfsense.ip.flags}|%{PF_SPEC:pfsense.ip.flags}),%{INT:network.iana_number},%{WORD:network.transport},",
"PF_IPv6_SPECIFIC_DATA": "(?(6)),%{BASE16NUM:pfsense.ip.tos},%{WORD:pfsense.ip.flow_label},%{WORD:pfsense.ip.flags},%{WORD:network.transport},%{INT:network.iana_number},",
"PF_PROTOCOL_DATA": "%{PF_TCP_DATA}|%{PF_UDP_DATA}|%{PF_ICMP_DATA}|%{PF_IGMP_DATA}|%{PF_IPv6_VAR}|%{PF_IPv6_ICMP}",
"PF_IPv6_VAR": "%{GREEDYDATA}",
"PF_IPv6_ICMP": "",
"PF_TCP_DATA": "%{INT:source.port:long},%{INT:destination.port:long},%{NONNEGINT:pfsense.tcp.length:long},%{WORD:pfsense.tcp.flags}?,%{NONNEGINT:pfsense.tcp.seq:long}?:?%{NONNEGINT},%{NONNEGINT:pfsense.tcp.ack:long}?,%{NONNEGINT:pfsense.tcp.window:long}?,%{WORD:pfsense.tcp.urg}?,%{GREEDYDATA:pfsense.tcp.options}",
"PF_UDP_DATA": "%{INT:source.port:long},%{INT:destination.port:long},%{NONNEGINT:pfsense.udp.length:long}$",
"PF_IGMP_DATA": "datalength=%{NONNEGINT:network.packets:long}",
"PF_ICMP_DATA": "%{PF_ICMP_TYPE}%{PF_ICMP_RESPONSE}",
"PF_ICMP_TYPE": "(?(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),",
"PF_ICMP_RESPONSE": "%{PF_ICMP_ECHO_REQ_REPLY}|%{PF_ICMP_UNREACHPORT}|%{PF_ICMP_UNREACHPROTO}|%{PF_ICMP_UNREACHABLE}|%{PF_ICMP_NEED_FLAG}|%{PF_ICMP_TSTAMP}|%{PF_ICMP_TSTAMP_REPLY}",
"PF_ICMP_ECHO_REQ_REPLY": "%{NONNEGINT:pfsense.icmp.id:long},%{NONNEGINT:pfsense.icmp.seq:long}",
"PF_ICMP_UNREACHPORT": "\\[?%{IP:pfsense.icmp.destination.ip}\\]?,\\[?%{WORD:pfsense.icmp.unreachable.protocol_id}\\]?,\\[?%{NONNEGINT:pfsense.icmp.unreachable.port:long}\\]?",
"PF_ICMP_UNREACHPROTO": "\\[?%{IP:pfsense.icmp.destination.ip}\\]?,\\[?%{WORD:pfsense.icmp.unreachable.protocol_id}\\]?",
"PF_ICMP_UNREACHABLE": "%{GREEDYDATA:pfsense.icmp.unreachable.other}",
"PF_ICMP_NEED_FLAG": "%{IP:pfsense.icmp.destination.ip},%{NONNEGINT:pfsense.icmp.mtu:long}",
"PF_ICMP_TSTAMP": "%{INT:pfsense.icmp.id},%{INT:pfsense.icmp.seq}",
"PF_ICMP_TSTAMP_REPLY": "%{INT:pfsense.icmp.id},%{INT:pfsense.icmp.seq},%{INT:pfsense.icmp.otime},%{INT:pfsense.icmp.rtime},%{INT:pfsense.icmp.ttime}",
"PF_SPEC": "[+]"
}
}
},
{
"set": {
"field": "event.kind",
"value": "event"
}
},
{
"set": {
"field": "event.action",
"value": "match",
"if": "ctx.event?.action == 'unkn(%u)'"
}
},
{
"append": {
"field": "event.type",
"value": "connection",
"allow_duplicates": false,
"if": "ctx.source?.address != null && ctx.destination?.address != null"
}
},
{
"append": {
"field": "event.type",
"value": "denied",
"allow_duplicates": false,
"if": "ctx.event.action == 'block'"
}
},
{
"append": {
"field": "event.type",
"value": "allowed",
"allow_duplicates": false,
"if": "ctx.event.action == 'pass'"
}
},
{
"lowercase": {
"field": "network.transport",
"ignore_missing": true
}
},
{
"remove": {
"field": "ack_number",
"ignore_missing": true,
"if": "ctx.ack_number == null || ctx.ack_number == ''"
}
},
{
"network_direction": {
"internal_networks_field": "_tmp.internal_networks"
}
},
{
"split": {
"field": "pfsense.tcp.options",
"separator": ";",
"ignore_missing": true,
"ignore_failure": true
}
},
{
"date": {
"field": "pfsense.icmp.otime",
"ignore_failure": true,
"formats": [
"UNIX",
"UNIX_MS"
]
}
},
{
"date": {
"field": "pfsense.icmp.rtime",
"ignore_failure": true,
"formats": [
"UNIX",
"UNIX_MS"
]
}
},
{
"date": {
"field": "pfsense.icmp.ttime",
"ignore_failure": true,
"formats": [
"UNIX",
"UNIX_MS"
]
}
}
]
### What did you see?
Hi,
In the pfsense 1.19.1 integration, the firewall ingest pipeline cannot properly parse logs which generates by pfsense floating rules with action `Match`.
Below are examples of syslog message logging generated by
Firewall rule with action Pass:
`163,,,1745940852,vtnet1.50,match,pass,in,4,0x0,,128,32369,0,none,1,icmp,60,10.100.4.10,1.1.1.1,request,3,1804340`
Firewall rule with action Reject:
`163,,,1745940852,vtnet1.50,match,block,in,4,0x0,,128,32361,0,none,1,icmp,60,10.100.4.10,1.1.1.1,request,3,1803240`
Firewall rule with action Block:
`163,,,1745940852,vtnet1.50,match,block,in,4,0x0,,128,32357,0,none,1,icmp,60,10.100.4.10,1.1.1.1,request,3,1802840`
Firewall rule with action Match:
`163,,,1745940852,vtnet1.50,match,unkn(%u),in,4,0x0,,128,32365,0,none,1,icmp,60,10.100.4.10,1.1.1.1,request,3,1803940`
where the seventh field contains the “Action” from the firewall rule, and for the rule with “Action=Match” in the log we see unkn(%u) and firewall ingest pipeline logs-pfsense.log-1.19.1-firewal is not designed to handle such a value.
here is the pattern_definitions part responsible for parsing log data:
`"PF_LOG_DATA": "%{INT},%{INT}?,,%{WORD:rule.id},%{DATA:observer.ingress.interface.name},%{PF_REASON:event.reason},%{WORD:event.action},%{WORD:network.direction},",`
where %{WORD:event.action} is directly responsible for parsing the 7th field and is not designed to receive non-WORD characters.
On my server I changed this pattern to `“PF_LOG_DATA”:“%{INT},%{INT}?,,%{WORD:rule.id},%{DATA:observer.ingress.interface.name},%{PF_REASON:event.reason},(?[^,]+),%{WORD:network.direction},”,`
But since this pipeline is part of the integration, these changes will be lost with each update.
Also this problem is seen more in the fact that pfsense generates not logical value of the field action=unkn(%u) instead of, for example, action=match. In this case the pipeline would work correctly. I will try to open a ticket in pfsense support, maybe they will be able to fix this problem.
### What did you expect to see?
Parsing logs containing Action=unkn(%u), possibly replacing with “unkn(%u)” with “match”
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.