Azure / Azure/azureml-examples

Machine Learning model for attack pattern on network logs

Open
#2,987 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Operating System

Windows

### Version Information

**Current situation:** I have **Network log (FortiAnalyzer log)** data and **endpoint log (SentinelOne)**

**sample logs for fortianalyzer**

<189>logver=700130566 timestamp=1705663428 devname="F" devid=6 vd=root date=2024-01-19 time=11:23:48 eventtime=1705659828815774000 tz=+0100 logid="0000000013" type=traffic subtype=forward level=notice srcip=40 srcport=42489 srcintf=Server-50 srcintfrole=lan dstip=********236 dstport=514 dstintf=wan2 dstintfrole=wan srcuuid=0e9b-51eb-cc6d-9f3b180f91fc dstuuid=8da***********ea-d8f0-c1ac2598a319 srccountry=Reserved dstcountry=Netherlands sessionid=568400008 proto=17 action=deny policyid=61 policytype=policy poluuid=9ec6b-e64e-7eee61e47eab policyname=jkhdkfjhk-LAN->iNetCatchAll service=SYSLOG trandisp=noop duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 vwlid=3 vwlquality=Seq_num(1 wan2), alive, selected vwlname=LAN-Rule appcat=unscanned crscore=30 craction=131072 crlevel=high srchwvendor=HP devtype=Network srcfamily=Switch srchwversion=Aruba mastersrcmac=d0::cc srcmac=d0:cc srcserver=0 <189>logver=700130566 timestamp=1705663428 devname="FGT0F" devid=FG13696 vd=root date=2024-01-19 time=11:23:48 eventtime=1705659828815830080 tz=+0100 logid="0000000013" type=traffic subtype=forward level=notice srcip=1040 srcport=42489 srcintf=Server-50 srcintfrole=lan dstip=1******.236 dstport=514 dstintf=wan2 dstintfrole=wan srcuuid=0eb*********-cc6d-9f3b180f91fc dstuuid=81ea-d8f0-c1ac2598a319 srccountry=Reserved dstcountry=China sessionid=568400009 proto=17 action=deny policyid=61 policytype=policy poluuid=9eceb-e64e-7eee61e47eab policyname=-LAN->iNetCatchAll service=SYSLOG trandisp=noop duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 vwlid=3 vwlquality=Seq_num(1 wan2), alive, selected vwlname=LAN-Rule appcat=unscanned crscore=30 craction=131072 crlevel=high srchwvendor=HP devtype=Network srcfamily=Switch srchwversion=Aruba mastersrcmac=d0:69:e8:cc srcmac=d0:6***:e8:cc srcserver=0

**sample logs of sentinel one**

<12>2023-11-07 12:34:45,230 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|19|New active threat - machine W023|1|osName=Windows 10 Enterprise rt=2023-11-07 12:30:36.456951 fileHash=3395856c**642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt cat=SystemEvent activityID=1813041897368528281 activityType=19 siteId=1080466588931492808 siteName=Technology . accountId=10815591 accountName= Technology . notificationScope=SITE <14>2023-11-07 12:34:45,231 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|2001|Kill performed successfully|1|fileHash=3395ee72602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt osName=Windows 10 Enterprise ip=2194 cat=SystemEvent suser=WP23 rt=#arcsightDate(Tue, 07 Nov 2023, 12:30:36 UTC) activityID=181305492937 activityType=2001 siteId=10804661492808 siteName= Technology . accountId=108046591 accountName= Technology . notificationScope=SITE <14>2023-11-07 12:34:45,231 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|2004|Quarantine performed successfully|1|fileHash=332b7382dee72602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt osName=Windows 10 Enterprise ip=2194 cat=SystemEvent suser=WP6023 rt=#arcsightDate(Tue, 07 Nov 2023, 12:30:36 UTC) activityID=1813045002 activityType=2004 siteId=10804665808 siteName= Technology . accountId=108015591 accountName= Technology . notificationScope=SITE <12>2023-11-07 12:34:45,415 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|19|New active threat - machine WP*******3|1|osName=Windows 10 Enterprise rt=2023-11-07 12:30:36.456951 fileHash=3395602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt cat=SystemEvent activityID=18130428281 activityType=19 siteId=108092808 siteName= Technology . accountId=1080466588914715591 accountName= Technology . notificationScope=SITE

**Problem statement: I would like to**

1. **connect (Stitch)** there two logs
2. **Apply an ML model to detect the anomalies** (for ex: DDOS, breach) on these logs

Summary:
**Is there any model on Huggingface.co** or other sources using which I can **detect the anomalies in the logs**?
also, **what is the best way to stitch the data** from these two logs

### Steps to reproduce

**Current situation:** I have **Network log (FortiAnalyzer log)** data and **endpoint log (SentinelOne)**

**sample logs for fortianalyzer**

<189>logver=700130566 timestamp=1705663428 devname="F" devid=6 vd=root date=2024-01-19 time=11:23:48 eventtime=1705659828815774000 tz=+0100 logid="0000000013" type=traffic subtype=forward level=notice srcip=40 srcport=42489 srcintf=Server-50 srcintfrole=lan dstip=********236 dstport=514 dstintf=wan2 dstintfrole=wan srcuuid=0e9b-51eb-cc6d-9f3b180f91fc dstuuid=8da***********ea-d8f0-c1ac2598a319 srccountry=Reserved dstcountry=Netherlands sessionid=568400008 proto=17 action=deny policyid=61 policytype=policy poluuid=9ec6b-e64e-7eee61e47eab policyname=jkhdkfjhk-LAN->iNetCatchAll service=SYSLOG trandisp=noop duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 vwlid=3 vwlquality=Seq_num(1 wan2), alive, selected vwlname=LAN-Rule appcat=unscanned crscore=30 craction=131072 crlevel=high srchwvendor=HP devtype=Network srcfamily=Switch srchwversion=Aruba mastersrcmac=d0::cc srcmac=d0:cc srcserver=0 <189>logver=700130566 timestamp=1705663428 devname="FGT0F" devid=FG13696 vd=root date=2024-01-19 time=11:23:48 eventtime=1705659828815830080 tz=+0100 logid="0000000013" type=traffic subtype=forward level=notice srcip=1040 srcport=42489 srcintf=Server-50 srcintfrole=lan dstip=1******.236 dstport=514 dstintf=wan2 dstintfrole=wan srcuuid=0eb*********-cc6d-9f3b180f91fc dstuuid=81ea-d8f0-c1ac2598a319 srccountry=Reserved dstcountry=China sessionid=568400009 proto=17 action=deny policyid=61 policytype=policy poluuid=9eceb-e64e-7eee61e47eab policyname=-LAN->iNetCatchAll service=SYSLOG trandisp=noop duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 vwlid=3 vwlquality=Seq_num(1 wan2), alive, selected vwlname=LAN-Rule appcat=unscanned crscore=30 craction=131072 crlevel=high srchwvendor=HP devtype=Network srcfamily=Switch srchwversion=Aruba mastersrcmac=d0:69:e8:cc srcmac=d0:6***:e8:cc srcserver=0

**sample logs of sentinel one**

<12>2023-11-07 12:34:45,230 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|19|New active threat - machine W023|1|osName=Windows 10 Enterprise rt=2023-11-07 12:30:36.456951 fileHash=3395856c**642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt cat=SystemEvent activityID=1813041897368528281 activityType=19 siteId=1080466588931492808 siteName=Technology . accountId=10815591 accountName= Technology . notificationScope=SITE <14>2023-11-07 12:34:45,231 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|2001|Kill performed successfully|1|fileHash=3395ee72602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt osName=Windows 10 Enterprise ip=2194 cat=SystemEvent suser=WP23 rt=#arcsightDate(Tue, 07 Nov 2023, 12:30:36 UTC) activityID=181305492937 activityType=2001 siteId=10804661492808 siteName= Technology . accountId=108046591 accountName= Technology . notificationScope=SITE <14>2023-11-07 12:34:45,231 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|2004|Quarantine performed successfully|1|fileHash=332b7382dee72602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt osName=Windows 10 Enterprise ip=2194 cat=SystemEvent suser=WP6023 rt=#arcsightDate(Tue, 07 Nov 2023, 12:30:36 UTC) activityID=1813045002 activityType=2004 siteId=10804665808 siteName= Technology . accountId=108015591 accountName= Technology . notificationScope=SITE <12>2023-11-07 12:34:45,415 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|19|New active threat - machine WP*******3|1|osName=Windows 10 Enterprise rt=2023-11-07 12:30:36.456951 fileHash=3395602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt cat=SystemEvent activityID=18130428281 activityType=19 siteId=108092808 siteName= Technology . accountId=1080466588914715591 accountName= Technology . notificationScope=SITE

**Problem statement: I would like to**

1. **connect (Stitch)** there two logs
2. **Apply an ML model to detect the anomalies** (for ex: DDOS, breach) on these logs

Summary:
**Is there any model on Huggingface.co** or other sources using which I can **detect the anomalies in the logs**?
also, **what is the best way to stitch the data** from these two logs

### Expected behavior

Model should detect the anomalies in the network log

### Actual behavior

Model should detect the anomalies in the network log

### Addition information

_No response_

Contributor guide

Open the contributing guide

Research direction

No repository file, test, or entry point is named. Start by reviewing the FortiAnalyzer and SentinelOne sample formats and narrowing the requested data stitching and anomaly-detection scope; done would require a specified model workflow and an example detection result.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, huggingface, machine-learning
Domain
data-engineering, machine-learning, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.