elastic / elastic/integrations
[pfSense] Squid logs from OPNsense are never parsed (ECS-JSON vs classic Squid format)
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Summary
The `pfsense` integration is documented and widely used for **OPNsense** as well as pfSense.
For every other provider (`filterlog`, `unbound`, `haproxy`, …) this works. For **Squid it
does not**: OPNsense emits Squid access logs as **ECS-JSON**, while
`data_stream/log/elasticsearch/ingest_pipeline/squid.yml` only has a `grok` pattern for the
classic Squid text format.
The result is silent: documents arrive, the grok fails, the pipeline `on_failure` fires, and
the event lands in `logs-pfsense.log-*` **unparsed** with `event.kind: pipeline_error`. Volume
dashboards look healthy, so nothing signals that proxy visibility is missing.
## Why this cannot be fixed on the OPNsense side
OPNsense hardcodes the Squid log format. In `/usr/local/etc/squid/squid.conf` it generates:
```
logformat opnsense {ECS-JSON…}
access_log syslog:local4.info opnsense
```
Both values of the GUI's *target* setting (`syslog` and `syslog_json`) produce the **identical**
`access_log` directive, and the file is regenerated by `configctl`, so an operator cannot
switch to classic Squid format from the UI. Any OPNsense user who enables the proxy and ships
logs with this integration has broken Squid parsing that they cannot fix in configuration.
## Reproduce
Send a Squid access log line from OPNsense, e.g.
```
<166>Aug 21 22:08:13 myfirewall.my-domain.tld (squid-1)[6802]: {"@timestamp":"2026-08-08T19:42:51+0000","ecs":{"version":"1.0.0"},"service":{"name":"proxy","type":"squid"},"host":{"hostname":"192.168.2.2"},"http":{"request":{"method":"GET"},"response":{"bytes":3162,"body":{"status_code":403}},"version":"1.1"},"source":{},"event":{"duration":"40","dataset":"squid.access"},"message":"GET http://example.com/ HTTP/1.1","user":{},"url":{"original":"http://example.com/"},"user_agent":{},"labels":{"request_status":"TCP_DENIED","hierarchy_status":"HIER_NONE"}}
```
**Expected:** `url.original`, `http.request.method`, `http.response.status_code`,
`squid.request_status` etc. populated.
**Actual:** no fields extracted; `event.kind: pipeline_error`, `error.message` set by the
pipeline's `on_failure`, and `tags: ["preserve_original_event"]`.
## Impact
* Squid is often the **only** source that records the true destination of proxied traffic —
endpoint and firewall telemetry only ever see the client→proxy hop. Losing it removes the
single source that can answer "where did this actually connect to".
* The integration's Squid content and any rule or query keyed on `url.original` /
`http.response.status_code` returns nothing for OPNsense users.
* Silent failure mode: byte volume looks normal, so the gap is only found by going looking.
## Proposed fix
Add a conditional JSON branch to `squid.yml` that decodes the OPNsense payload and maps it
onto the **same fields the existing grok branch produces**, leaving the classic path untouched.
One implementation detail worth recording, because it is a trap: the decode **must not** use
`add_to_root`. The OPNsense payload contains its own `event` object, which overwrites
`event.provider`. Since `default.yml` routes on `ctx.event.provider == 'squid'` and drops
anything whose provider is not in the allowlist, merging to root causes documents to be
silently dropped.
Happy to open a PR — I have a patch validated against both formats, including a differential
test showing the classic pfSense fixtures produce byte-identical output before and after.
### Environment
* Integration: `pfsense` 1.25.2 (behaviour identical on current `main` — `squid.yml`
unchanged since 2025-11-06)
* Elastic Stack: 9.5.0
* Source: OPNsense 26.1 (FreeBSD 14.3), Squid with `sslbump`
Contributor guide
Research direction
Start with data_stream/log/elasticsearch/ingest_pipeline/squid.yml and compare its classic grok branch with the OPNsense ECS-JSON example. Read default.yml to verify provider routing, then use the mentioned classic pfSense fixtures and differential test approach. Done means OPNsense fields such as url.original, http.request.method, http.response.status_code, and squid.request_status are populated while classic output remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100