elastic / elastic/integrations

[pfsense] Parsing errors when rule action is Match

Open
#13,738 3 comments 0 reactions 0 assignees View on GitHub
Integration:pfsense maintainer:Community needs:triage Team:Security-Deployment and Devices
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

pfSense [pfsense]

### Dataset Name

pfsense.log

### Integration Version

1.19.1

### Agent Version

8.14.3

### Agent Output Type

elasticsearch

### Elasticsearch Version

8.14.3

### OS Version and Architecture

Security Onion 2.4.100 (Oracle Linux Server 9.5)

### Software/API Version

_No response_

### Error Message

_No response_

### Event Original

_No response_

### What did you do?

logs-pfsense.log-1.19.1-firewall (fixed):
[
{
"grok": {
"field": "message",
"patterns": [
"%{PF_LOG_ENTRY}%{GREEDYDATA}"
],
"pattern_definitions": {
"PF_LOG_ENTRY": "%{PF_LOG_DATA}%{PF_IP_SPECIFIC_DATA}%{PF_IP_DATA}%{PF_PROTOCOL_DATA}?",
"PF_LOG_DATA": "%{INT},%{INT}?,,%{WORD:rule.id},%{DATA:observer.ingress.interface.name},%{PF_REASON:event.reason},(?[^,]+),%{WORD:network.direction},",
"PF_REASON": "[a-zA-Z-]+",
"PF_IP_DATA": "%{NONNEGINT:network.bytes:long},%{IP:source.address},%{IP:destination.address},",
"PF_IP_SPECIFIC_DATA": "%{PF_IPv4_SPECIFIC_DATA}|%{PF_IPv6_SPECIFIC_DATA}",
"PF_IPv4_SPECIFIC_DATA": "(?(4)),%{BASE16NUM:pfsense.ip.tos},%{WORD:pfsense.ip.ecn}?,%{NONNEGINT:pfsense.ip.ttl:long},%{NONNEGINT:pfsense.ip.id:long},%{NONNEGINT:pfsense.ip.offset:long},(?:%{WORD:pfsense.ip.flags}|%{PF_SPEC:pfsense.ip.flags}),%{INT:network.iana_number},%{WORD:network.transport},",
"PF_IPv6_SPECIFIC_DATA": "(?(6)),%{BASE16NUM:pfsense.ip.tos},%{WORD:pfsense.ip.flow_label},%{WORD:pfsense.ip.flags},%{WORD:network.transport},%{INT:network.iana_number},",
"PF_PROTOCOL_DATA": "%{PF_TCP_DATA}|%{PF_UDP_DATA}|%{PF_ICMP_DATA}|%{PF_IGMP_DATA}|%{PF_IPv6_VAR}|%{PF_IPv6_ICMP}",
"PF_IPv6_VAR": "%{GREEDYDATA}",
"PF_IPv6_ICMP": "",
"PF_TCP_DATA": "%{INT:source.port:long},%{INT:destination.port:long},%{NONNEGINT:pfsense.tcp.length:long},%{WORD:pfsense.tcp.flags}?,%{NONNEGINT:pfsense.tcp.seq:long}?:?%{NONNEGINT},%{NONNEGINT:pfsense.tcp.ack:long}?,%{NONNEGINT:pfsense.tcp.window:long}?,%{WORD:pfsense.tcp.urg}?,%{GREEDYDATA:pfsense.tcp.options}",
"PF_UDP_DATA": "%{INT:source.port:long},%{INT:destination.port:long},%{NONNEGINT:pfsense.udp.length:long}$",
"PF_IGMP_DATA": "datalength=%{NONNEGINT:network.packets:long}",
"PF_ICMP_DATA": "%{PF_ICMP_TYPE}%{PF_ICMP_RESPONSE}",
"PF_ICMP_TYPE": "(?(request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),",
"PF_ICMP_RESPONSE": "%{PF_ICMP_ECHO_REQ_REPLY}|%{PF_ICMP_UNREACHPORT}|%{PF_ICMP_UNREACHPROTO}|%{PF_ICMP_UNREACHABLE}|%{PF_ICMP_NEED_FLAG}|%{PF_ICMP_TSTAMP}|%{PF_ICMP_TSTAMP_REPLY}",
"PF_ICMP_ECHO_REQ_REPLY": "%{NONNEGINT:pfsense.icmp.id:long},%{NONNEGINT:pfsense.icmp.seq:long}",
"PF_ICMP_UNREACHPORT": "\\[?%{IP:pfsense.icmp.destination.ip}\\]?,\\[?%{WORD:pfsense.icmp.unreachable.protocol_id}\\]?,\\[?%{NONNEGINT:pfsense.icmp.unreachable.port:long}\\]?",
"PF_ICMP_UNREACHPROTO": "\\[?%{IP:pfsense.icmp.destination.ip}\\]?,\\[?%{WORD:pfsense.icmp.unreachable.protocol_id}\\]?",
"PF_ICMP_UNREACHABLE": "%{GREEDYDATA:pfsense.icmp.unreachable.other}",
"PF_ICMP_NEED_FLAG": "%{IP:pfsense.icmp.destination.ip},%{NONNEGINT:pfsense.icmp.mtu:long}",
"PF_ICMP_TSTAMP": "%{INT:pfsense.icmp.id},%{INT:pfsense.icmp.seq}",
"PF_ICMP_TSTAMP_REPLY": "%{INT:pfsense.icmp.id},%{INT:pfsense.icmp.seq},%{INT:pfsense.icmp.otime},%{INT:pfsense.icmp.rtime},%{INT:pfsense.icmp.ttime}",
"PF_SPEC": "[+]"
}
}
},
{
"set": {
"field": "event.kind",
"value": "event"
}
},
{
"set": {
"field": "event.action",
"value": "match",
"if": "ctx.event?.action == 'unkn(%u)'"
}
},
{
"append": {
"field": "event.type",
"value": "connection",
"allow_duplicates": false,
"if": "ctx.source?.address != null && ctx.destination?.address != null"
}
},
{
"append": {
"field": "event.type",
"value": "denied",
"allow_duplicates": false,
"if": "ctx.event.action == 'block'"
}
},
{
"append": {
"field": "event.type",
"value": "allowed",
"allow_duplicates": false,
"if": "ctx.event.action == 'pass'"
}
},
{
"lowercase": {
"field": "network.transport",
"ignore_missing": true
}
},
{
"remove": {
"field": "ack_number",
"ignore_missing": true,
"if": "ctx.ack_number == null || ctx.ack_number == ''"
}
},
{
"network_direction": {
"internal_networks_field": "_tmp.internal_networks"
}
},
{
"split": {
"field": "pfsense.tcp.options",
"separator": ";",
"ignore_missing": true,
"ignore_failure": true
}
},
{
"date": {
"field": "pfsense.icmp.otime",
"ignore_failure": true,
"formats": [
"UNIX",
"UNIX_MS"
]
}
},
{
"date": {
"field": "pfsense.icmp.rtime",
"ignore_failure": true,
"formats": [
"UNIX",
"UNIX_MS"
]
}
},
{
"date": {
"field": "pfsense.icmp.ttime",
"ignore_failure": true,
"formats": [
"UNIX",
"UNIX_MS"
]
}
}
]

### What did you see?

Hi,

In the pfsense 1.19.1 integration, the firewall ingest pipeline cannot properly parse logs which generates by pfsense floating rules with action `Match`.

Below are examples of syslog message logging generated by
Firewall rule with action Pass:
`163,,,1745940852,vtnet1.50,match,pass,in,4,0x0,,128,32369,0,none,1,icmp,60,10.100.4.10,1.1.1.1,request,3,1804340`
Firewall rule with action Reject:
`163,,,1745940852,vtnet1.50,match,block,in,4,0x0,,128,32361,0,none,1,icmp,60,10.100.4.10,1.1.1.1,request,3,1803240`
Firewall rule with action Block:
`163,,,1745940852,vtnet1.50,match,block,in,4,0x0,,128,32357,0,none,1,icmp,60,10.100.4.10,1.1.1.1,request,3,1802840`
Firewall rule with action Match:
`163,,,1745940852,vtnet1.50,match,unkn(%u),in,4,0x0,,128,32365,0,none,1,icmp,60,10.100.4.10,1.1.1.1,request,3,1803940`
where the seventh field contains the “Action” from the firewall rule, and for the rule with “Action=Match” in the log we see unkn(%u) and firewall ingest pipeline logs-pfsense.log-1.19.1-firewal is not designed to handle such a value.

here is the pattern_definitions part responsible for parsing log data:
`"PF_LOG_DATA": "%{INT},%{INT}?,,%{WORD:rule.id},%{DATA:observer.ingress.interface.name},%{PF_REASON:event.reason},%{WORD:event.action},%{WORD:network.direction},",`
where %{WORD:event.action} is directly responsible for parsing the 7th field and is not designed to receive non-WORD characters.
On my server I changed this pattern to `“PF_LOG_DATA”:“%{INT},%{INT}?,,%{WORD:rule.id},%{DATA:observer.ingress.interface.name},%{PF_REASON:event.reason},(?[^,]+),%{WORD:network.direction},”,`
But since this pipeline is part of the integration, these changes will be lost with each update.
Also this problem is seen more in the fact that pfsense generates not logical value of the field action=unkn(%u) instead of, for example, action=match. In this case the pipeline would work correctly. I will try to open a ticket in pfsense support, maybe they will be able to fix this problem.

### What did you expect to see?

Parsing logs containing Action=unkn(%u), possibly replacing with “unkn(%u)” with “match”

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.