elastic / elastic/integrations
[iis]: iis.error grok can silently misparse HTTPERR lines with extra columns
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
IIS [packages/iis]
### Dataset Name
`iis.error`
### Integration Version
1.24.9 (also reproduced on 1.24.7)
### Agent Version
N/A (reproduced at the ingest-pipeline level via `elastic-package test pipeline`)
### Agent Output Type
elasticsearch
### Elasticsearch Version
9.5.0
### OS Version and Architecture
Windows Server 2019 / 2022, x86_64 (HTTP.sys HTTPERR logs)
### Software/API Version
HTTP.sys error logging (`httperr*.log`)
### Error Message
None on the failing path. There is no `_grokparsefailure` and no `error.message`. The document is indexed as a successful parse.
### Event Original
Representative HTTPERR log lines (synthetic values):
Longer line (16 space-separated columns):
```
2026-01-15 12:00:00 192.0.2.10 50000 192.0.2.20 443 HTTP/1.1 GET / - - 503 1 N/A DefaultAppPool TCP
```
Shorter line that the current pipeline handles (14 columns):
```
2026-01-15 12:00:00 192.0.2.10 50000 192.0.2.20 80 HTTP/1.1 GET / - 503 1 N/A DefaultAppPool
```
### What did you do?
Ran these lines through the `iis.error` ingest pipeline (`packages/iis/data_stream/error/elasticsearch/ingest_pipeline/default.yml`) using `elastic-package test pipeline`.
### What did you see?
The 16-column line matches a shorter grok pattern and is treated as success. Values land in the wrong fields and the HTTP status code is missing:
- `http.response.status_code`: missing (the `503` is not captured)
- `iis.error.reason_phrase`: `1` (wrong)
- `iis.error.queue_name`: `N/A` (wrong)
- trailing tokens (`DefaultAppPool`, `TCP`): dropped
- no parse-failure tag
The 14-column line parses as expected (`http.response.status_code: 503`).
The grok patterns are not end-anchored (no trailing `$`). The leading `TIMESTAMP_ISO8601` fixes the start of the match, but with no end anchor a line with extra trailing columns still matches a shorter pattern. The unmatched columns are simply ignored and the match is reported as success.
HTTPERR column count is not a single fixed layout. HTTP.sys field selection is configurable (`ErrorLoggingFields`), and newer Windows versions can emit additional standard fields (queue name, stream id, transport, and others). Unselected fields are omitted from the line, so length varies. 16 columns is one valid layout, not the maximum; a "log everything" `#Fields` line can be 20+ columns.
Related: the pipeline test added in #20273 includes a line that ends with `TCP` and expects `iis.error.queue_name: "TCP"`. That looks like the same class of misparse baked into a fixture.
Current `_dev/test/pipeline` coverage is the shorter layouts only.
### What did you expect to see?
- The pipeline should correctly parse the common documented HTTPERR layouts, following the fixed HTTP.sys field order, including the modern longer layouts (for example the 16-column layout that appends `s-siteid`, a stream id, and a trailing transport value).
- Standard fields should map to the correct ECS fields on every supported layout: `http.response.status_code` present, reason/queue not shifted, `s-siteid` captured (it is a default HTTP.sys field but is currently matched and discarded), and `transport` lowercased into `network.transport` (ECS requires lowercase).
- Every grok pattern should be end-anchored so that a line whose layout the pipeline does not recognize fails visibly (`on_failure`) instead of silently matching a shorter pattern and indexing wrong values.
### Anything else?
This is a parser correctness issue in `iis.error`, not an agent or ingest-node failure. The `error` stream can look healthy while documents are wrong.
Background on why this needs a range of layouts, not a single new pattern: HTTPERR fields are written in a fixed order, but which fields appear is a per-server `ErrorLoggingFields` bitmask, and newer Windows versions append fields (stream id, transport). Disabled fields are omitted entirely (so column count shifts), while enabled-but-empty fields are logged as `-`. So the same estate can legitimately produce 13-, 14-, and 16-column lines.
Suggested direction (covering the realistic range of layouts, not just the one that failed):
1. **End-anchor all existing patterns.** General correctness fix. Stops silent misparse for every column count the pipeline does not recognize. Verified against the existing pipeline tests: currently-parsing layouts are byte-for-byte unchanged, and longer/unknown lines (16- and 22-column tested) switch from silent misparse to a visible `error.message`.
2. **Cover the common layouts** by following the fixed field order with anchored patterns: the documented 13-column default, the 14-column variant, and the modern 16-column layout that appends `s-siteid` / stream id / transport. Capture `s-siteid` (add `iis.error.site_id` to `data_stream/error/fields/fields.yml`) and lowercase `transport` into `network.transport`.
3. **Explicit boundary (handling 16+ fields).** Arbitrary `ErrorLoggingFields` combinations — especially optional middle fields such as user-agent / cookie / referer / host / bytes / time-taken inserted before the status code — cannot be disambiguated from a single line, because two different configurations can produce the same column count with different field meanings. Those are intentionally left to fail closed. Fully general parsing would require the `#Fields:` header, which the ingest pipeline never sees (it is stripped by `exclude_lines: ["^#"]`, and pipelines process one event at a time).
4. **Fix and extend the fixtures.** Add `_dev/test/pipeline` coverage for the 13/14/16-column layouts, and correct the fixture added in #20273 (the trailing `TCP` is a transport value, not `queue_name`).
5. `iis.access` is positional and unanchored in the same way and is worth a separate look, but is out of scope for this issue.
Contributor guide
Research direction
Start with packages/iis/data_stream/error/elasticsearch/ingest_pipeline/default.yml and the existing _dev/test/pipeline fixtures, then run them with elastic-package test pipeline. End-anchor the existing patterns, add coverage for the 13-, 14-, and 16-column layouts, capture s-siteid in data_stream/error/fields/fields.yml, and map transport to network.transport. Done means unknown layouts fail visibly, supported layouts retain correct fields, and the #20273 fixture no longer treats TCP as queue_name.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100