elastic / elastic/integrations

[cisco_ise]: kv processor discards the whole continuation-segment payload, leaving log_details unset

Open
#21,006 2 comments 0 reactions 0 assignees View on GitHub
bug Integration:cisco_ise needs:triage Team:Integration-Experience
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

Note: This issue has been drafted with :robot: Cursor/Claude Opus 5 under my supervision.

## Summary

When a Cisco ISE syslog message is split across segments, the continuation segment can begin in the middle of a value. The `kv` processor that populates `cisco_ise.log.log_details` then rejects the **entire** payload, so every key/value pair in that segment is discarded and `cisco_ise.log.log_details` is never created.

Three patch releases have now fixed a different downstream consumer of that missing field, rather than the `kv` that fails to populate it:

| version | entry |
|---|---|
| `1.32.11` | Fix NullPointerException when evaluating bracket access on absent `cisco_ise.log.log_details` (Failed_Attempts foreach/`User-Name` appends, Authentication Flow Diagnostics `NAS-IP-Address` convert, RADIUS Accounting `Event-Timestamp` date) |
| `1.32.12` | Handle empty continuation segments in CISE_Administrative_and_Operational_Audit logs |
| `1.32.14` | Add null check for `script_parse_av_pairs` processor (#20976) |

Each of those is correct in isolation. None of them recovers the dropped fields, and after #20976 the loss is no longer visible in the document at all.

## Why the `kv` fails

All 16 category pipelines share the same processor shape, e.g. `packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_passed_authentications.yml` line 190:

```yaml
- kv:
tag: kv_cisco_ise_log_log_details_raw_to_cisco_ise_log_log_details_dd59a8ac
field: cisco_ise.log.log_details_raw
target_field: cisco_ise.log.log_details
field_split: ', (?=[^,=]+=)'
value_split: =
ignore_failure: true
```

For a continuation segment the payload starts mid-value, so `field_split` yields a first token with no `=` — `dd0011` in the fixture below. Elasticsearch's `KeyValueProcessor` throws for any part missing `value_split`, and because the processor carries `ignore_failure: true` with no `on_failure`, the whole processor is skipped and `target_field` is never written. One malformed leading token costs all of the well-formed pairs behind it.

## Impact

Measured with `_simulate` against the published `1.32.13` and `1.32.14` packages. Field counts are leaf fields in `_source`, excluding `message`, `event.original`, `ecs.*`, `@timestamp`, `tags` and `error.*`. "stripped" is the identical line with only the leading partial token removed:

| fixture line | `1.32.13` as-is | `1.32.14` as-is | `1.32.14` stripped |
|---|---|---|---|
| `test-pipeline-passed-authentications.log:22` | `pipeline_error`, 8 fields | `event`, 8 fields | `event`, **78 fields** |
| `test-pipeline-failed-attempts.log:40` | `event`, 9 fields | `event`, 9 fields | `event`, **23 fields** |
| `test-pipeline-radius-accounting.log:5` | `event`, 10 fields | `event`, 10 fields | `event`, **14 fields** |
| `test-pipeline-authentication-flow-diagnostics.log:11` | `event`, 10 fields | `event`, 10 fields | `event`, **15 fields** |

The Passed Authentications fragment carries 53 distinct keys / 83 pairs and keeps 8 fields. Recovered by stripping the leading token: `user.name`, `related.user`, `related.ip`, `client.mac`, `cisco_ise.log.nas.ip`, `cisco_ise.log.cpm.session.id`, the full `cisco_ise.log.cisco_av_pair.*` tree that `script_parse_av_pairs` exists to build, and the whole `Response={...}` block.

Two consequences:

1. **The loss is now silent for Passed Authentications.** Before #20976 the document carried `event.kind: pipeline_error` and an `error.message`, which was noisy but at least queryable. After it, a document missing ~70 fields is indistinguishable from a fully parsed one. The other three categories in the table were already silent and are unchanged by #20976.
2. **More consumers of `log_details` will keep surfacing.** The field is read by dozens of `rename`/`convert`/`append` processors per pipeline; the ones that happen to use bracket access or a script are the ones that have needed patches so far.

## How the numbers were measured

Every count above came from running the committed fixture line through the real Fleet pipeline of a published package version, rather than from reading the YAML. The runs used log-shape, an internal service that installs a given package version on a simulation cluster and reports what its pipeline did to one line. By hand the equivalent is to install the package and call `POST _ingest/pipeline/_simulate?verbose` against `logs-cisco_ise.log--pipeline_passed_authentications` with `{"@timestamp": "...", "message": ""}`.

Three variants were run per fixture line:

1. the line exactly as committed, against `1.32.13` (the parent of #20976)
2. the same line against `1.32.14` (#20976, published)
3. the same line with only the leading partial token removed, against `1.32.14` — a stand-in for what `kv` would receive if that token were stripped before it

The first reproduces the failure verbatim:

```
event.kind = pipeline_error
error.message = Processor 'script' with tag 'script_parse_av_pairs' in pipeline
'logs-cisco_ise.log-1.32.13-pipeline_passed_authentications' failed with
message 'cannot access method/field [get] from a null def reference'
```

The second shows that error gone and no field gained. The third shows what is recoverable. For the three non-Passed-Authentications rows, variants 1 and 2 are identical, which is how we know #20976 does not change them.

Two limits worth stating, since they bound what the table proves:

- The service simulates versions **published to EPR**, so it cannot run a modified working tree. The `1.32.14 stripped` column therefore proves that the remaining pairs parse and which fields they reach, but it does not exercise the suggested `gsub` or its `if` condition — those still need confirming on a stack.
- Counts are leaf fields in `_source` after the excludes listed above, so a multi-value array such as `cisco_ise.log.step` counts once.

The 223-payload gate check under *Suggested fix* is a separate and weaker kind of evidence: the `kv` `field_split` regex modelled in Python over every `log_details_raw` payload reachable from the package's pipeline fixtures, to see where a candidate `if` condition would fire. That one is static analysis, not a cluster run.

## Suggested fix

Drop the leading partial value before the `kv`, gated so it only fires when the payload does not already start with a key:

```yaml
- gsub:
tag: gsub_strip_leading_partial_value_9c1f0a3d
description: >-
Continuation segments can begin in the middle of a value, leaving a
leading token with no '='. The kv processor below rejects the entire
payload in that case, so remove the partial token first.
field: cisco_ise.log.log_details_raw
pattern: '^.*?, (?=[^,=]+=)'
replacement: ''
if: >-
ctx.cisco_ise?.log?.log_details_raw != null &&
!(ctx.cisco_ise.log.log_details_raw =~ /^[^,=]+=/)
ignore_missing: true
```

Checked against all 223 `cisco_ise.log.log_details_raw` payloads reachable from the package's own pipeline fixtures: the gate fires on exactly the 5 that currently fail and skips the other 218, so no currently-working event changes shape.

Since this belongs in all 16 pipelines that parse `log_details_raw`, a shared pipeline is probably the durable form rather than 16 copies. Whatever the shape, it would help to make any residual loss visible instead of silent:

```yaml
- append:
tag: append_tag_log_details_unparsed
description: >-
Tag events whose log_details_raw could not be parsed into key/value
pairs so the unparsed payloads can be audited.
field: tags
value: cisco_ise.log_details_unparsed
allow_duplicates: false
if: ctx.cisco_ise?.log?.log_details_raw != null && ctx.cisco_ise?.log?.log_details == null
```

## Reproducing

The fixtures are already in the repo. `packages/cisco_ise/data_stream/log/_dev/test/pipeline/test-pipeline-passed-authentications.log` line 22 is a segment `1/2` continuation whose payload begins `dd0011, NAS-IP-Address=10.30.1.1, ...`; its committed expectation contains only `cisco_ise.log.category.name`, `cisco_ise.log.message.id`, `cisco_ise.log.segment.*`, `host.hostname`, `log.syslog.priority` and `related.hosts`. Removing `dd0011, ` from that line and re-simulating produces the 78-field document.

`test-pipeline-failed-attempts.log:40`, `test-pipeline-radius-accounting.log:5` and `test-pipeline-authentication-flow-diagnostics.log:11` are the same shape (payload begins `alice.johnson, StepData=...`), and their committed expectations likewise carry no `cisco_ise.log.log_details.*`.

Contributor guide

Open the contributing guide

Research direction

Start with packages/cisco_ise/data_stream/log/elasticsearch/ingest_pipeline/pipeline_passed_authentications.yml and the continuation fixture at packages/cisco_ise/data_stream/log/_dev/test/pipeline/test-pipeline-passed-authentications.log:22. Compare the other three named fixture lines and inspect how all 16 pipelines process log_details_raw. Done means continuation payloads retain valid key/value pairs without changing the 218 already-working payloads, with the pipeline fixtures confirming the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, yaml
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.