elastic / elastic/integrations
Missing pipeline parameter in generated agent policy for unifiedlogs-based datasets (macOS Security Events package)
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
### Integration Name
macOS Security Events [packages/macos]
### Dataset Name
all datasets included in this integration
### Integration Version
1.0.0
### Agent Version
9.4.3
### Agent Output Type
elasticsearch
### Elasticsearch Version
9.4.3
### OS Version and Architecture
ECK
### Software/API Version
_No response_
### Error Message
no error
### Event Original
{"timezoneName":"","eventType":"activityCreateEvent","formatString":"","userID":0,"activityIdentifier":"REDACTED","subsystem":"","category":"","threadID":"REDACTED","senderImageUUID":"REDACTED","backtrace":{"frames":[{"imageOffset":1476680,"imageUUID":"REDACTED"}]},"bootUUID":"","processImagePath":"/Applications/Bitdefender/CoreSecurity.app/Contents/Daemons/BDCoreIssues.bundle/Contents/MacOS/BDCoreIssues","senderImagePath":"/System/Library/Frameworks/Security.framework/Versions/A/Security","timestamp":"2026-07-17T14:52:03Z","creatorActivityID":0,"machTimestamp":"REDACTED","eventMessage":"SecTrustEvaluateIfNecessary","processImageUUID":"REDACTED","traceID":"REDACTED","processID":"REDACTED","senderProgramCounter":1476680,"parentActivityIdentifier":0}
### What did you do?
- Installed the macOS Security Events integration on a Fleet-managed 9.4.3 stack.
- Enabled the Authentication, System changes, and Account management log categories in an integration policy, and deployed it to an enrolled macOS agent.
- Let logs flow in, then inspected the resulting documents in logs-macos.authentication-*, logs-macos.system_changes-*, and logs-macos.account_management-*. and see that the logs were not parsed
- Confirmed the dedicated ingest pipeline for each dataset works correctly in isolation with `POST _ingest/pipeline/logs-macos.authentication-1.0.0/_simulate`
- Pulled the full compiled agent policy (Fleet > Agent policies > policy > View policy) and inspected the unifiedlogs input stanza for each affected dataset and see they were no pipeline attribute
### What did you see?
- The _simulate calls against logs-macos.authentication-1.0.0 (and the other affected pipelines) correctly returned enriched documents, e.g. event.category: authentication added by the first processor - confirming the pipelines themselves are valid and functional.
- The compiled unifiedlogs input stanza for Authentication, System changes, and Account management contained no pipeline key at all.
- The corresponding backing indices had no index.default_pipeline / index.final_pipeline set either (empty response), so there was no fallback mechanism applying the pipeline.
- As a result, all documents ingested through these three datasets were stored completely unprocessed: no event.category, no ECS enrichment, none of the documented macos.* fields - everything remained in message / event.original only. This affected all data ingested since the integration was installed.
- After manually adding the pipeline: override in the advanced YAML for each affected stream, redeploying, and re-checking, documents ingested afterward were correctly parsed and enriched - confirming the pipelines are fine and the issue is purely in how the policy is generated for these datasets.
### What did you expect to see?
The compiled unifiedlogs input stanza for each dataset should include a pipeline key pointing at the dataset's dedicated ingest pipeline by default, without requiring a manual override, e.g.:
Even if it appears in the GUI, it's not effective in the real exported policy
### Anything else?
_No response_
Contributor guide
Research direction
Start in packages/macos and inspect the unifiedlogs input stanzas for the Authentication, System changes, and Account management datasets in the compiled Fleet policy. Compare them with the dedicated pipelines using POST _ingest/pipeline/logs-macos.authentication-1.0.0/_simulate. Done means generated policies include each dataset's pipeline by default and newly ingested documents contain the expected parsed and enriched fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, macos
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100