Azure / Azure/azureml-examples
Machine Learning model for attack pattern on network logs
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 1.7k
- Avg merge
- 18h 18m
- Merged PRs (30d)
- 2
Description
### Operating System
Windows
### Version Information
**Current situation:** I have **Network log (FortiAnalyzer log)** data and **endpoint log (SentinelOne)**
**sample logs for fortianalyzer**
<189>logver=700130566 timestamp=1705663428 devname="F" devid=6 vd=root date=2024-01-19 time=11:23:48 eventtime=1705659828815774000 tz=+0100 logid="0000000013" type=traffic subtype=forward level=notice srcip=40 srcport=42489 srcintf=Server-50 srcintfrole=lan dstip=********236 dstport=514 dstintf=wan2 dstintfrole=wan srcuuid=0e9b-51eb-cc6d-9f3b180f91fc dstuuid=8da***********ea-d8f0-c1ac2598a319 srccountry=Reserved dstcountry=Netherlands sessionid=568400008 proto=17 action=deny policyid=61 policytype=policy poluuid=9ec6b-e64e-7eee61e47eab policyname=jkhdkfjhk-LAN->iNetCatchAll service=SYSLOG trandisp=noop duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 vwlid=3 vwlquality=Seq_num(1 wan2), alive, selected vwlname=LAN-Rule appcat=unscanned crscore=30 craction=131072 crlevel=high srchwvendor=HP devtype=Network srcfamily=Switch srchwversion=Aruba mastersrcmac=d0::cc srcmac=d0:cc srcserver=0 <189>logver=700130566 timestamp=1705663428 devname="FGT0F" devid=FG13696 vd=root date=2024-01-19 time=11:23:48 eventtime=1705659828815830080 tz=+0100 logid="0000000013" type=traffic subtype=forward level=notice srcip=1040 srcport=42489 srcintf=Server-50 srcintfrole=lan dstip=1******.236 dstport=514 dstintf=wan2 dstintfrole=wan srcuuid=0eb*********-cc6d-9f3b180f91fc dstuuid=81ea-d8f0-c1ac2598a319 srccountry=Reserved dstcountry=China sessionid=568400009 proto=17 action=deny policyid=61 policytype=policy poluuid=9eceb-e64e-7eee61e47eab policyname=-LAN->iNetCatchAll service=SYSLOG trandisp=noop duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 vwlid=3 vwlquality=Seq_num(1 wan2), alive, selected vwlname=LAN-Rule appcat=unscanned crscore=30 craction=131072 crlevel=high srchwvendor=HP devtype=Network srcfamily=Switch srchwversion=Aruba mastersrcmac=d0:69:e8:cc srcmac=d0:6***:e8:cc srcserver=0
**sample logs of sentinel one**
<12>2023-11-07 12:34:45,230 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|19|New active threat - machine W023|1|osName=Windows 10 Enterprise rt=2023-11-07 12:30:36.456951 fileHash=3395856c**642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt cat=SystemEvent activityID=1813041897368528281 activityType=19 siteId=1080466588931492808 siteName=Technology . accountId=10815591 accountName= Technology . notificationScope=SITE <14>2023-11-07 12:34:45,231 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|2001|Kill performed successfully|1|fileHash=3395ee72602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt osName=Windows 10 Enterprise ip=2194 cat=SystemEvent suser=WP23 rt=#arcsightDate(Tue, 07 Nov 2023, 12:30:36 UTC) activityID=181305492937 activityType=2001 siteId=10804661492808 siteName= Technology . accountId=108046591 accountName= Technology . notificationScope=SITE <14>2023-11-07 12:34:45,231 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|2004|Quarantine performed successfully|1|fileHash=332b7382dee72602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt osName=Windows 10 Enterprise ip=2194 cat=SystemEvent suser=WP6023 rt=#arcsightDate(Tue, 07 Nov 2023, 12:30:36 UTC) activityID=1813045002 activityType=2004 siteId=10804665808 siteName= Technology . accountId=108015591 accountName= Technology . notificationScope=SITE <12>2023-11-07 12:34:45,415 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|19|New active threat - machine WP*******3|1|osName=Windows 10 Enterprise rt=2023-11-07 12:30:36.456951 fileHash=3395602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt cat=SystemEvent activityID=18130428281 activityType=19 siteId=108092808 siteName= Technology . accountId=1080466588914715591 accountName= Technology . notificationScope=SITE
**Problem statement: I would like to**
1. **connect (Stitch)** there two logs
2. **Apply an ML model to detect the anomalies** (for ex: DDOS, breach) on these logs
Summary:
**Is there any model on Huggingface.co** or other sources using which I can **detect the anomalies in the logs**?
also, **what is the best way to stitch the data** from these two logs
### Steps to reproduce
**Current situation:** I have **Network log (FortiAnalyzer log)** data and **endpoint log (SentinelOne)**
**sample logs for fortianalyzer**
<189>logver=700130566 timestamp=1705663428 devname="F" devid=6 vd=root date=2024-01-19 time=11:23:48 eventtime=1705659828815774000 tz=+0100 logid="0000000013" type=traffic subtype=forward level=notice srcip=40 srcport=42489 srcintf=Server-50 srcintfrole=lan dstip=********236 dstport=514 dstintf=wan2 dstintfrole=wan srcuuid=0e9b-51eb-cc6d-9f3b180f91fc dstuuid=8da***********ea-d8f0-c1ac2598a319 srccountry=Reserved dstcountry=Netherlands sessionid=568400008 proto=17 action=deny policyid=61 policytype=policy poluuid=9ec6b-e64e-7eee61e47eab policyname=jkhdkfjhk-LAN->iNetCatchAll service=SYSLOG trandisp=noop duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 vwlid=3 vwlquality=Seq_num(1 wan2), alive, selected vwlname=LAN-Rule appcat=unscanned crscore=30 craction=131072 crlevel=high srchwvendor=HP devtype=Network srcfamily=Switch srchwversion=Aruba mastersrcmac=d0::cc srcmac=d0:cc srcserver=0 <189>logver=700130566 timestamp=1705663428 devname="FGT0F" devid=FG13696 vd=root date=2024-01-19 time=11:23:48 eventtime=1705659828815830080 tz=+0100 logid="0000000013" type=traffic subtype=forward level=notice srcip=1040 srcport=42489 srcintf=Server-50 srcintfrole=lan dstip=1******.236 dstport=514 dstintf=wan2 dstintfrole=wan srcuuid=0eb*********-cc6d-9f3b180f91fc dstuuid=81ea-d8f0-c1ac2598a319 srccountry=Reserved dstcountry=China sessionid=568400009 proto=17 action=deny policyid=61 policytype=policy poluuid=9eceb-e64e-7eee61e47eab policyname=-LAN->iNetCatchAll service=SYSLOG trandisp=noop duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 rcvdpkt=0 vwlid=3 vwlquality=Seq_num(1 wan2), alive, selected vwlname=LAN-Rule appcat=unscanned crscore=30 craction=131072 crlevel=high srchwvendor=HP devtype=Network srcfamily=Switch srchwversion=Aruba mastersrcmac=d0:69:e8:cc srcmac=d0:6***:e8:cc srcserver=0
**sample logs of sentinel one**
<12>2023-11-07 12:34:45,230 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|19|New active threat - machine W023|1|osName=Windows 10 Enterprise rt=2023-11-07 12:30:36.456951 fileHash=3395856c**642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt cat=SystemEvent activityID=1813041897368528281 activityType=19 siteId=1080466588931492808 siteName=Technology . accountId=10815591 accountName= Technology . notificationScope=SITE <14>2023-11-07 12:34:45,231 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|2001|Kill performed successfully|1|fileHash=3395ee72602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt osName=Windows 10 Enterprise ip=2194 cat=SystemEvent suser=WP23 rt=#arcsightDate(Tue, 07 Nov 2023, 12:30:36 UTC) activityID=181305492937 activityType=2001 siteId=10804661492808 siteName= Technology . accountId=108046591 accountName= Technology . notificationScope=SITE <14>2023-11-07 12:34:45,231 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|2004|Quarantine performed successfully|1|fileHash=332b7382dee72602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt osName=Windows 10 Enterprise ip=2194 cat=SystemEvent suser=WP6023 rt=#arcsightDate(Tue, 07 Nov 2023, 12:30:36 UTC) activityID=1813045002 activityType=2004 siteId=10804665808 siteName= Technology . accountId=108015591 accountName= Technology . notificationScope=SITE <12>2023-11-07 12:34:45,415 sentinel - CEF:0|SentinelOne|Mgmt|Y#20|19|New active threat - machine WP*******3|1|osName=Windows 10 Enterprise rt=2023-11-07 12:30:36.456951 fileHash=3395602f798b642f14140 filePath=\Device\HarddiskVolume3\TEMP\virus.txt cat=SystemEvent activityID=18130428281 activityType=19 siteId=108092808 siteName= Technology . accountId=1080466588914715591 accountName= Technology . notificationScope=SITE
**Problem statement: I would like to**
1. **connect (Stitch)** there two logs
2. **Apply an ML model to detect the anomalies** (for ex: DDOS, breach) on these logs
Summary:
**Is there any model on Huggingface.co** or other sources using which I can **detect the anomalies in the logs**?
also, **what is the best way to stitch the data** from these two logs
### Expected behavior
Model should detect the anomalies in the network log
### Actual behavior
Model should detect the anomalies in the network log
### Addition information
_No response_
Contributor guide
Research direction
No repository file, test, or entry point is named. Start by reviewing the FortiAnalyzer and SentinelOne sample formats and narrowing the requested data stitching and anomaly-detection scope; done would require a specified model workflow and an example detection result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, huggingface, machine-learning
- Domain
- data-engineering, machine-learning, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100