03: Security Monitoring and Analysis¶
In this session we will cover:
- What is a Security Information and Event Management (SIEM) tool?
- What data sources may we use with a SIEM?
- Hands-on with Splunk and the "Boss of the SOC"
- Cyber Ranges
Security Information and Event Management¶
SIEM Tools¶
Security Information and Event Management (SIEM) is software that improves security awareness of an IT environment by combining security information management (SIM) and security event management (SEM). SIEM solutions enhance threat detection, compliance, and security incident management through the gathering and analysis of real-time and historical security event data and sources. SIEM main capabilities provide a broad range of log event collection and management, increasing the ability to analyze log events and other data across dissimilar sources, and operational capabilities including incident management, dashboards, and reporting. SIEM also offers data aggregation across the enterprise network and normalization of that data for further analysis. Additionally, SIEM helps enable security monitoring, user activity monitoring, and compliance.
Splunk¶
Splunk is a widely-used SIEM platform. It essentially provides the mechanism to ingest and search data at scale, meaning that it is well-suited for machine data such as machine-generated logging. The use case of Splunk actually extends beyond cyber security to various other big data analysis domains. Users can search data using the Splunk Query Language, a similar concept to that of Structured Query Language used for common databases.
Alternative SIEMs¶
Formerly known as ELK (elasticsearch, logstash, kibana) elastic is another popular SIEM platform. Others include greylog, LogPoint, and SecurityOnion.
... and even Jupyter SIEM integration!¶
Microsoft Threat Intelligence Center (MSTIC) - a prime example of where Jupyter is being used for security investigations - thanks to their MSTICpy and MSTICnb libraries. Notebooks are a fantastic way to extend capability of a SIEM and ensuring repeatability of playbooks.
Yet More Data Sources...¶
Suricata¶
Suricata is an open-source Intrusion Detection System (IDS). 'Suricata is the leading independent open source threat detection engine. By combining intrusion detection (IDS), intrusion prevention (IPS), network security monitoring (NSM) and PCAP processing, Suricata can quickly identify, stop, and assess the most sophisticated attacks'.
- Logging is captured in the
/var/log/suricata/eve.jsonfile (event file). - Useful guide for setting up Suricata
import pandas as pd
df = pd.read_json('./data/eve.json', lines=True)
print (df['event_type'].unique())
df[df['event_type']=='http'].head(5)
['dns' 'stats' 'http' 'flow' 'dhcp' 'fileinfo' 'tls' 'alert']
| timestamp | flow_id | in_iface | event_type | src_ip | src_port | dest_ip | dest_port | proto | dns | ... | dhcp | tcp | fileinfo | tls | alert | icmp_type | icmp_code | response_icmp_type | response_icmp_code | metadata | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3 | 2021-11-05 14:29:39.860519+00:00 | 6.998918e+14 | ens33 | http | 192.168.147.146 | 42046.0 | 35.224.170.84 | 80.0 | TCP | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 41 | 2021-11-05 14:31:53.262235+00:00 | 6.875953e+14 | ens33 | http | 192.168.147.147 | 50414.0 | 35.232.111.17 | 80.0 | TCP | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 97 | 2021-11-05 14:34:09.967206+00:00 | 1.865670e+15 | ens33 | http | 192.168.147.148 | 47998.0 | 35.224.170.84 | 80.0 | TCP | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 105 | 2021-11-05 14:34:32.908273+00:00 | 1.219845e+15 | ens33 | http | 192.168.147.146 | 38268.0 | 35.232.111.17 | 80.0 | TCP | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 167 | 2021-11-05 14:37:13.473230+00:00 | 1.779694e+15 | ens33 | http | 192.168.147.147 | 50418.0 | 35.232.111.17 | 80.0 | TCP | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
5 rows × 25 columns
#data = !osqueryi "select * from os_version"
import json
data = !osqueryi --json "select * from os_version"
d = data.get_list()
for i in d:
print (i)
W0211 11:53:46.102381 26064 options.cpp:101] Cannot set unknown or invalid flag: utc
[
{"arch":"64-bit","build":"19044","codename":"Microsoft Windows 10 Education","install_date":"1615324060","major":"10","minor":"0","name":"Microsoft Windows 10 Education","patch":"","platform":"windows","platform_like":"windows","version":"10.0.19044"}
]
Sysmon¶
System Monitor (Sysmon) - Windows system service to monitor and log system activity to the Windows event log.
- Detailed information about process creations, network connections, and changes to file creation time.
- Identify malicious or anomalous activity and understand how intruders and malware operate on your network.
Note that Sysmon does not provide analysis of the events it generates, nor does it attempt to protect or hide itself from attackers.
Active Directory¶
Active Directory - directory service used by Windows domain networks to assign permissions/role-based access controls with integration of network file and resource share management, largely inspired by the earlier Lightweight Directory Access Protocol (LDAP).
Cyber Range¶
What is a Cyber Range?¶
A cyber range is a means to simulate attack and defense on a realistic (simulated) computing infrastructure. In its most basic form we can use two virtual machines to convey attack and target - a cyber range extends this to have an instrumented lab with monitoring and defense response.
DetectionLab¶
DetectionLab is developed by Chris Long, Senior Analyst at Netflix, and offers a quick and easy way to create your own virtualised cyber range. It incorporates a number of the tools used today - Splunk, Suricata, osquery, Active Directory, Microsoft ATA, and more. It manages log generation and collection. It has Active Directory built in. This is a perfect set up for testing offensive and defensive security measures against a realistic virtualised infrastructure, without taking hours (or days!) to set up.
Splunk Attack Range and Security Content¶
Splunk Attack Range is a similar concept, and a more recent cyber range (2021) for deploying a small lab environment with the ability to conduct offensive testing (e.g., Atomic Red Team), and capturing log activities to detect the threat.
The Splunk Security Content project gives you access to the Splunk repository of Analytic Stories, security guides that provide background on tactics, techniques and procedures (TTPs), mapped to the MITRE ATT&CK Framework, the Lockheed Martin Cyber Kill Chain, and CIS Controls. This include Splunk searches, machine learning algorithms and Splunk Phantom playbooks (where available)—all designed to work together to detect, investigate, and respond to threats.
Cyber Range¶
We will look at these in more detail in the coming weeks... you will use them for your Portfolio Task 3...
Hands-on with Splunk¶
- Have a look at the blog post about Boss of the SOC (BOTS).
- The first BOTS challenge is freely available to use - we can use it via TryHackMe or via SamsClass
- Task 2 requires the Splunk Quick Reference Guide that describes key concepts and features, as well as commonly used commands and functions.
- Take a closer look at Task 4 - you may want to continue this in your lab sessions.
Splunk in Practice - Reconnaissance¶
- Access via SamsClass or via TryHackMe
What IP is scanning our web server? (imreallynotbatman.com)
index=botsv1 imreallynotbatman.com | stats count by src_ipindex=botsv1 imreallynotbatman.com | stats count by sourcetypeindex=botsv1 imreallynotbatman.com sourcetype="stream:htttp" | stats count by src_ip
95% of the traffic is associated with the address 40.80.148.42
What web scanner scnnned the server?
index=botsv1 src=40.80.148.42 sourcetype=stream:http | stats count by src_headersindex=botsv1 src=40.80.148.42 sourcetype=stream:http | stats count by http_user_agent
Amongst some odd data, we see a name Acunetix.
What is the IP address of our web server?
index=botsv1 src=40.80.148.42 sourcetype=stream:http | stats count by dst
We find that the majority of activity is associated with 192.168.250.70
What content management system is imreallynotbatman.com using?
index=botsv1 dest=192.168.250.70 sourcetype=stream:http | stats count by uri | sort - count
We find that the records with the highest count refer to joomla.
Now over to you...¶
Practical Session¶
- Carry on with Portfolio Task 1
- Examine the Splunk Boss of the SOC practical exercise
- Continue to explore the lab examples:
- 03-MSTICpy Tutorial