opensearch-project / opensearch-project/security-analytics
[FEATURE] Store raw field mappings in dedicated system index instead of storing as field-aliases in customer index
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 111
- Forks
- 111
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 13
Description
Is your feature request related to a problem?
Currently in Security Analytics, customer's log data indices' field names are mapped to field names as understood by ecs or ocsf schema naming for all the log types and field aliases are created in the customer's log indices
Detector rules are then converted into opensearch queries. Queries have the field names as present in config and field aliases will then help refer back the the raw field name in the index during searches
Current behaviour makes immutable changes to user's log data which is not a desirable design
What solution would you like?
- Store mappings of raw field to rule field in system index instead of as field aliases
- Create rules with raw field names by replacing the config field names by using the mappings.
For eg. if user has an index on which he wish to create a cloudtrail detector with 1 rule.
Index mappings before creating detector
{
"ct": {
"mappings": {
"properties": {
"en": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"es": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
Cloudtrail Sigma Rule
id: abae8fec-57bd-4f87-aff6-6e3db989843d
logsource:
product: cloudtrail
title: AWS Snapshot Backup Exfiltration
description: >-
Detects the modification of an EC2 snapshot's permissions to enable access
from another account
tags:
- attack.exfiltration
- attack.t1537
falsepositives:
- Valid change to a snapshot's permissions
level: medium
status: test
references:
- 'https://www.justice.gov/file/1080281/download'
- 'https://attack.mitre.org/techniques/T1537/'
author: Darin Smith
detection:
selection_source:
eventSource: ec2.amazonaws.com
eventName: ModifySnapshotAttribute
condition: selection_source
Index mappings after creating detector with cloud trail rule
{
"ct": {
"mappings": {
"properties": {
"aws": {
"properties": {
"cloudtrail": {
"properties": {
"event_name": {
"type": "alias",
"path": "en"
},
"event_source": {
"type": "alias",
"path": "es"
}
}
}
}
},
"en": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"es": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
Why we need new approach
- Current impl is not a correct experience because it would fail detector execution flow if newer version changes the name of the alias for the log type.
- If user is using an index pattern or data stream where they change the name of the field that would lead to more aliases i.e. more customer data manipulation.
- Change alias name would be a breaking change in new version requiring user to delete old data and re-create indices.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by tracing the existing field-alias creation and detector query-conversion flow described in the issue. Done means raw-to-rule mappings are stored in a system index, detector rules use raw field names, and customer log indices are not modified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100