elastic / elastic/integrations
[cisco_ise]: kv processor discards the whole continuation-segment payload, leaving log_details unset
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
Note: This issue has been drafted with :robot: Cursor/Claude Opus 5 under my supervision.
## Summary
When a Cisco ISE syslog message is split across segments, the continuation segment can begin in the middle of a value. The `kv` processor that populates `cisco_ise.log.log_details` then rejects the **entire** payload, so every key/value pair in that segment is discarded and `cisco_ise.log.log_details` is never created.
Three patch releases have now fixed a different downstream consumer of that missing field, rather than the `kv` that fails to populate it:
| version | entry |
|---|---|
| `1.32.11` | Fix NullPointerException when evaluating bracket access on absent `cisco_ise.log.log_details` (Failed_Attempts foreach/`User-Name` appends, Authentication Flow Diagnostics `NAS-IP-Address` convert, RADIUS Accounting `Event-Timestamp` date) |
| `1.32.12` | Handle empty continuation segments in CISE_Administrative_and_Operational_Audit logs |
| `1.32.14` | Add null check for `script_parse_av_pairs` processor (#20976) |
Each of those is correct in isolation. None of them recovers the dropped fields, and after #20976 the loss is no longer visible in the document at all.
## Why the `kv` fails
All 16 category pipelines share the same processor shape, e.g. `packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_passed_authentications.yml` line 190:
```yaml
- kv:
tag: kv_cisco_ise_log_log_details_raw_to_cisco_ise_log_log_details_dd59a8ac
field: cisco_ise.log.log_details_raw
target_field: cisco_ise.log.log_details
field_split: ', (?=[^,=]+=)'
value_split: =
ignore_failure: true
```
For a continuation segment the payload starts mid-value, so `field_split` yields a first token with no `=` — `dd0011` in the fixture below. Elasticsearch's `KeyValueProcessor` throws for any part missing `value_split`, and because the processor carries `ignore_failure: true` with no `on_failure`, the whole processor is skipped and `target_field` is never written. One malformed leading token costs all of the well-formed pairs behind it.
## Impact
Measured with `_simulate` against the published `1.32.13` and `1.32.14` packages. Field counts are leaf fields in `_source`, excluding `message`, `event.original`, `ecs.*`, `@timestamp`, `tags` and `error.*`. "stripped" is the identical line with only the leading partial token removed:
| fixture line | `1.32.13` as-is | `1.32.14` as-is | `1.32.14` stripped |
|---|---|---|---|
| `test-pipeline-passed-authentications.log:22` | `pipeline_error`, 8 fields | `event`, 8 fields | `event`, **78 fields** |
| `test-pipeline-failed-attempts.log:40` | `event`, 9 fields | `event`, 9 fields | `event`, **23 fields** |
| `test-pipeline-radius-accounting.log:5` | `event`, 10 fields | `event`, 10 fields | `event`, **14 fields** |
| `test-pipeline-authentication-flow-diagnostics.log:11` | `event`, 10 fields | `event`, 10 fields | `event`, **15 fields** |
The Passed Authentications fragment carries 53 distinct keys / 83 pairs and keeps 8 fields. Recovered by stripping the leading token: `user.name`, `related.user`, `related.ip`, `client.mac`, `cisco_ise.log.nas.ip`, `cisco_ise.log.cpm.session.id`, the full `cisco_ise.log.cisco_av_pair.*` tree that `script_parse_av_pairs` exists to build, and the whole `Response={...}` block.
Two consequences:
1. **The loss is now silent for Passed Authentications.** Before #20976 the document carried `event.kind: pipeline_error` and an `error.message`, which was noisy but at least queryable. After it, a document missing ~70 fields is indistinguishable from a fully parsed one. The other three categories in the table were already silent and are unchanged by #20976.
2. **More consumers of `log_details` will keep surfacing.** The field is read by dozens of `rename`/`convert`/`append` processors per pipeline; the ones that happen to use bracket access or a script are the ones that have needed patches so far.
## How the numbers were measured
Every count above came from running the committed fixture line through the real Fleet pipeline of a published package version, rather than from reading the YAML. The runs used log-shape, an internal service that installs a given package version on a simulation cluster and reports what its pipeline did to one line. By hand the equivalent is to install the package and call `POST _ingest/pipeline/_simulate?verbose` against `logs-cisco_ise.log--pipeline_passed_authentications` with `{"@timestamp": "...", "message": ""}`.
Three variants were run per fixture line:
1. the line exactly as committed, against `1.32.13` (the parent of #20976)
2. the same line against `1.32.14` (#20976, published)
3. the same line with only the leading partial token removed, against `1.32.14` — a stand-in for what `kv` would receive if that token were stripped before it
The first reproduces the failure verbatim:
```
event.kind = pipeline_error
error.message = Processor 'script' with tag 'script_parse_av_pairs' in pipeline
'logs-cisco_ise.log-1.32.13-pipeline_passed_authentications' failed with
message 'cannot access method/field [get] from a null def reference'
```
The second shows that error gone and no field gained. The third shows what is recoverable. For the three non-Passed-Authentications rows, variants 1 and 2 are identical, which is how we know #20976 does not change them.
Two limits worth stating, since they bound what the table proves:
- The service simulates versions **published to EPR**, so it cannot run a modified working tree. The `1.32.14 stripped` column therefore proves that the remaining pairs parse and which fields they reach, but it does not exercise the suggested `gsub` or its `if` condition — those still need confirming on a stack.
- Counts are leaf fields in `_source` after the excludes listed above, so a multi-value array such as `cisco_ise.log.step` counts once.
The 223-payload gate check under *Suggested fix* is a separate and weaker kind of evidence: the `kv` `field_split` regex modelled in Python over every `log_details_raw` payload reachable from the package's pipeline fixtures, to see where a candidate `if` condition would fire. That one is static analysis, not a cluster run.
## Suggested fix
Drop the leading partial value before the `kv`, gated so it only fires when the payload does not already start with a key:
```yaml
- gsub:
tag: gsub_strip_leading_partial_value_9c1f0a3d
description: >-
Continuation segments can begin in the middle of a value, leaving a
leading token with no '='. The kv processor below rejects the entire
payload in that case, so remove the partial token first.
field: cisco_ise.log.log_details_raw
pattern: '^.*?, (?=[^,=]+=)'
replacement: ''
if: >-
ctx.cisco_ise?.log?.log_details_raw != null &&
!(ctx.cisco_ise.log.log_details_raw =~ /^[^,=]+=/)
ignore_missing: true
```
Checked against all 223 `cisco_ise.log.log_details_raw` payloads reachable from the package's own pipeline fixtures: the gate fires on exactly the 5 that currently fail and skips the other 218, so no currently-working event changes shape.
Since this belongs in all 16 pipelines that parse `log_details_raw`, a shared pipeline is probably the durable form rather than 16 copies. Whatever the shape, it would help to make any residual loss visible instead of silent:
```yaml
- append:
tag: append_tag_log_details_unparsed
description: >-
Tag events whose log_details_raw could not be parsed into key/value
pairs so the unparsed payloads can be audited.
field: tags
value: cisco_ise.log_details_unparsed
allow_duplicates: false
if: ctx.cisco_ise?.log?.log_details_raw != null && ctx.cisco_ise?.log?.log_details == null
```
## Reproducing
The fixtures are already in the repo. `packages/cisco_ise/data_stream/log/_dev/test/pipeline/test-pipeline-passed-authentications.log` line 22 is a segment `1/2` continuation whose payload begins `dd0011, NAS-IP-Address=10.30.1.1, ...`; its committed expectation contains only `cisco_ise.log.category.name`, `cisco_ise.log.message.id`, `cisco_ise.log.segment.*`, `host.hostname`, `log.syslog.priority` and `related.hosts`. Removing `dd0011, ` from that line and re-simulating produces the 78-field document.
`test-pipeline-failed-attempts.log:40`, `test-pipeline-radius-accounting.log:5` and `test-pipeline-authentication-flow-diagnostics.log:11` are the same shape (payload begins `alice.johnson, StepData=...`), and their committed expectations likewise carry no `cisco_ise.log.log_details.*`.
Contributor guide
Research direction
Start with packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_passed_authentications.yml and the continuation fixture at packages/cisco_ise/data_stream/log/_dev/test/pipeline/test-pipeline-passed-authentications.log:22. Compare the other three named fixture lines and inspect how all 16 pipelines process log_details_raw. Done means continuation payloads retain valid key/value pairs without changing the 218 already-working payloads, with the pipeline fixtures confirming the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, yaml
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100