Sumo Logic ahead of the pack
Read articleComplete visibility for DevSecOps
Reduce downtime and move from reactive to proactive monitoring.
September 28, 2023
For folks who are responsible for threat detection of any kind for their organizations, the cloud can often be a difficult area to approach.
At the time of writing, Amazon Web Services contains over two hundred services, while the Azure cloud offers six hundred.
Each of these services can generate unique telemetry and each surface can present defenders with a unique attack path to handle. Adding to this complexity is the diversity of cloud workload configurations, as well as varying architecture models.
How do we, as protectors of the enterprise, begin to tackle visibility and threat detection use cases for these types of environments?
In previous blog posts, we have outlined what purple teaming is and why it’s a powerful tool to bring to bear in your enterprise. Now, let’s continue this theme and cover how to apply the purple teaming concept to a Microsoft Entra/Microsoft Azure environment which consists of compute, identity and productivity components.
As noted in the introduction to this post, the Azure cloud is associated with hundreds of services.
Complicating this is the fact that the Azure cloud provides mechanisms for identity management via Microsoft Entra, compute options via Microsoft Azure, as well as various productivity services via Office 365.
Each of these three “pillars” has a unique set of telemetry and use cases tied to it, with various attack surfaces. In addition, from a threat actor point of view, the tradecraft necessary to meet objectives will no doubt differ depending on which component or service is being attacked or targeted.
In this context, a purple team can be a valuable exercise that can be executed on an Azure cloud environment.
It is difficult to start from scratch in this regard, so let’s briefly mind map the following broad areas of purple teaming in the context of the Azure cloud: planning, desired outcomes, areas of focus and finally, pitfalls.
Let’s apply this approach to a fictitious organization named FakeORG that utilizes the following Azure cloud services:
This organization believes they have the necessary telemetry for all the above services and workloads coming into their SIEM platform and want to purple team this environment in order to test their assumptions, discover gaps and create threat detection rules in order to wrangle malicious activity.
After examining existing incidents and looking through a few threat intel reports, the team decides that the following techniques will be executed during the purple team exercise:
These tactics, techniques and procedures cover identity, compute, containerized environments as well as developer paradigms and will provide a good starting point for future purple team efforts.
The participants of the purple team exercise have also agreed to keep notes on each of the executions, in the following format:
TTP | Replication Steps | Required Data Source | Existing Detection? |
– | – | – | – |
The team has now chosen which techniques to replicate in the environment, has confirmed that the necessary access is in place and has also settled on a note-taking format.
Now the fun begins! The chosen tactics, techniques and procedures need to be executed within the environment while an eye is kept on the relevant telemetry on the other end.
Password spraying is a critical area to wrap detective and preventive controls around, and FakeORG engineers have had to deal with a number of attacks on their environment that began with a password spray attempt.
It makes sense then, that our FakeORG security engineers have decided to execute this technique during their purple team exercise.
In order to perform this password spray attack, they will be using the popular CredMaster tool.
The execution of this technique will look something like:
In typical password spray attacks, FakeORG engineers would be looking for failed authentication requests to multiple accounts coming from the same source IP address.
However, the CredMaster tool throws them a bit of a curve ball:
“Launch a password spray / brute force attack via Amazon AWS passthrough proxies, shifting the requesting IP address for every authentication attempt. This dynamically creates FireProx APIs for more evasive password sprays.”
The key point here is that each authentication request has a new IP address associated with it, as a new AWS API Gateway is spun up dynamically.
It is quickly discovered that existing password spray detections fail to capture this activity, as these detections rely on the failed authentication requests stemming from a single IP address.
This is a great example of why purple teaming is an excellent tool for getting to the “ground truth” of threat detection posture. The FakeORG engineers have at this point discovered a bit of a gap in their detection.
The engineers brainstorm a little and discover that AWS provides a list of Gateway IP addresses, the engineers craft a cool one-liner to grab these IP addresses:
curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes[] | select(.service == "API_GATEWAY") | {ip_prefix} | join (" ")'
Following the relevant Cloud SIEM documentation, the engineers take the output of the above command and craft a CSV file with the following format:
and proceed to upload it to their Cloud SIEM instance:
Now, any record ingested into their Cloud SIEM instance that matches the IP range discovered above will be labeled with “AWS_API_Gateway” - the team can now utilize their logs and craft detections using the following logic:
In query form, this would look like:
_index=sec_record_authentication | where device_ip_location = "AWS_API_Gateway" | where !success | timeslice 10m | count_distinct(device_ip) as ip_addresses,values(user_username) as usernames, values(srcDevice_ip) as source_ips,values(application) as applications by _timeslice | where ip_addresses > 1
And the results:
During prototyping, it is noted that when timeslicing the data into 10 minute intervals, two authentication failures in each of those time blocks are observed stemming from the AWS API Gateway, which could be indicative of password spray attacks from tools that utilize the API Gateway for IP rotation purposes.
Moving on to the next test, the team at FakeORG have read many threat intelligence reports outlining attack flows whereby threat actors register a new multi-factor authentication device in Entra/Azure AD.
The test for this is relatively straightforward and it is decided to create a new temporary account, register an MFA device for the account and then browse to aka.ms/mfasetup and register a new MFA device for this account.
In this instance, an alert every time an MFA method is changed is deemed unnecessary. However, it is decided that this execution should be discovered in the logs, as this would be a critical aspect to check if an account were to bubble up as compromised.
The following query is crafted:
_index=sec_record_audit cloud_provider = "Azure" action = "Update user" | %"fields.modified_property_strongauthenticationphoneappdetail" as new_device | %"fields.properties.targetresources.1.modifiedproperties.1.oldvalue" as old_device | targetUser_username as username | where !isnull(new_device) | where !isnull(old_device) | json field=new_device "DeviceName" as new_device_name | json field=old_device "[0].DeviceName" as old_device_name | fields username,action,new_device_name,old_device_name
And the results are returned:
In this case, the same MFA device was removed and added back in order to simulate the activity, but the underlying query logic works and this search is saved in order to be performed as part of broader incident response processes.
For the Entra ID service, the notes are now:
TTP | Replication Steps | Required Data Source | Existing Detection? |
Password Spray Entra ID | Password spray via CredMaster | Azure AD / Entra ID Sign In Logs | No |
Password Spray Entra ID - 2nd Attempt | Password spray via Credmaster | Azure AD / Entra ID Sign In Logs | Detection crafted |
Register MFA Device | Browse to aka.ms/mfasetup, log in with the relevant user account and follow prompts to register a new multi-factor authentication device. | Azure AD / Entra ID Audit Logs, UserManagement Category | Real-time detection deemed unnecessary, search created and saved |
Very cool! The team can now start to see the purple team exercise taking shape and starting to deliver value, with three executions completed, a new detection being crafted and some searches created - the team keeps going and shifts gears a little bit towards Office 365
In this test, the team will attempt to determine if they can see when a user performs sensitive searches for documents found on SharePoint sites, and then proceeds to download these same sensitive documents.
For this test, the SnaffPoint tool will be utilized.
The tool is executed and sure enough, it finds a document containing the keyword search of “passwords” - a test account is utilized in order to view and download the document.
The team starts to dig through the available SharePoint telemetry and sees various operations pertaining to this file, including the file being accessed as well as the search query being performed.
In isolation, these events are interesting but do not paint a behavioral picture - something is needed to glue the events together in a logical fashion.
After some tinkering, it is decided that a qualifier-type query would probably be the best approach - the query itself ends up looking like:
_index=sec_record_audit resource=SharePoint | 0 as score | "" as messageQualifiers | "" as messageQualifiers1 | "" as messageQualifiers2 | "" as messageQualifiers3 | timeslice 1h | %"fields.SourceFileName" as source_file_name | contains(file_basename,source_file_name) as file_searchdl_compare | if (%"fields.searchquerytext" matches /(password)/, concat(messageQualifiers, "Sensitive SharePoint Search Performed: ",%"fields.searchquerytext", "\n# score: 300\n"),"") as messageQualifiers | if (action matches /(FileAccessed)/, concat(messageQualifiers1, "SharePoint File Access: ",source_file_name, "\n# score: 300\n"),"") as messageQualifiers1 | if (action matches /(FileDownloaded)/, concat(messageQualifiers2, "SharePoint File Downloaded: ",file_basename, "\n# score: 300\n"),"") as messageQualifiers2 | if(file_searchdl_compare, concat(messageQualifiers3, "File Search and Access Match","\n# score: 300\n"),"") as messageQualifiers3 | concat(messageQualifiers,messageQualifiers1,messageQualifiers2,messageQualifiers3) as q | parse regex field=q "score:\s(?<score>-?\d+)" multi | where !isEmpty(q) | values(q) as qualifiers,sum(score) as score by user_username,_timeslice
This query is:
The returned results are extremely interesting and look something like the following:
Here, we see two distinct results side by side, each broken down by the configured time slice, and the result on the left hand side has a score of 12,600 whereas the result on the right has a score of 18,600.
The reason why the event on the right side is scoring higher is that it contains a qualifier for the user in question actually downloading the file that was queried and accessed. In other words, the execution chain on the left side tells you that someone may have performed a SharePoint query containing sensitive keywords and may have accessed a file containing sensitive keywords or information, but did not download this file.
The team now has various options for tweaking these qualifiers to suit their environment and can run this search on a schedule to bubble up an alert when the score exceeds a certain threshold.
After looking at SharePoint, the team reads an article that describes an attack flow whereby a misconfigured Teams tenant could potentially allow guest users to message users within an organization and send files that could be used for phishing attacks.
Current configurations are checked and it is discovered that currently, the organization is not affected by this misconfiguration, and the Teams tenant has external access disabled. However, since this setting can be modified, an alert that looks for this modification is desired.
The operation is performed in the tenant, and the logs are monitored.
The following Cloud SIEM rule is crafted:
metadata_product = "Office 365" and action = "TeamsAdminAction" and lower(commandLine) matches /(tenantfederationsettings\/configuration\/global)/ and fields["ModifiedProperties.1.Name"] = "AllowFederatedUsers" and fields["ModifiedProperties.1.NewValue"] = true
Once the test is performed again, a relevant Signal is generated:
Now when a Teams organizational setting to allow external guest access is enabled, a Signal is generated that can be actioned and investigated.
Now the notes read:
TTP | Replication Steps | Required Data Source | Existing Detection? |
Password Spray Entra ID | Password spray via CredMaster | Azure AD / Entra ID Sign In Logs | No |
Password Spray Entra ID - 2nd Attempt | Password spray via Credmaster | Azure AD / Entra ID Sign In Logs | Detection crafted |
Register MFA Device | Browse to aka.ms/mfasetup, log in with the relevant user account and follow prompts to register a new multi factor authentication device. | Azure AD / Entra ID Audit Logs, UserManagement Category | Real-time detection deemed unnecessary, search created and saved |
SharePoint Exfiltration | Create test SharePoint site, run SnaffPoint to search for a sensitive file and then download the file | Office 365 logs, SharePoint operations | Detection Crafted |
Teams External Access Enabled | Browse to Office admin portal, enable external access to Teams | Office 365, TeamsAdminAction | No, CloudSIEM Signal created for enablement of external Teams guest access |
Prior to performing the purple team exercise, the team came across an awesome blog post regarding Azure virtual machine run commands and wanted to see if their environment had the necessary telemetry in place to detect this activity.
A test virtual machine is provisioned and a run command is issued to it. The command executes successfully, but no telemetry is visible related to this event.
The Azure AD diagnostic settings are double checked and see that everything seems to be enabled:
After some digging, it is discovered that there is another diagnostic setting that was missed.
Navigating to the Azure portal, then Monitor, then Activity log, the activity is visible there, but the engineers do not have these events forwarded to something like an event hub.
The team goes ahead and creates the new diagnostic setting, forwarding all activity to an event hub:
A run command is issued to the virtual machine again, and this time the necessary telemetry is present and a CloudSIEM rule is crafted:
metadata_vendor = 'Microsoft' AND metadata_product = 'Azure' AND action = "Administrative" AND description = "MICROSOFT.COMPUTE/VIRTUALMACHINES/RUNCOMMAND/ACTION"
With a signal being generated:
The team now moves from a run command to deleting a virtual machine, to ensure that they get some kind of signal or alert when a virtual machine is deleted in their environment, the team proceeds to create and delete a test virtual machine and everything works!
The team receives the following CloudSIEM signal:
A nice smooth test where everything worked as expected, the team is happy with this result and proceeds to update their notes:
TTP | Replication Steps | Required Data Source | Existing Detection? |
Password Spray Entra ID | Password spray via CredMaster | Azure AD / Entra ID Sign In Logs | No |
Password Spray Entra ID - 2nd Attempt | Password spray via Credmaster | Azure AD / Entra ID Sign In Logs | Detection crafted |
Register MFA Device | Browse to aka.ms/mfasetup, log in with the relevant user account and follow prompts to register a new multi-factor authentication device. | Azure AD / Entra ID Audit Logs, UserManagement Category | Real-time detection deemed unnecessary, search created and saved |
SharePoint Exfiltration | Create test SharePoint site, run SnaffPoint to search for a sensitive file and then download the file | Office 365 logs, SharePoint operations | Detection Crafted |
Teams External Access Enabled | Browse to Office admin portal, enable external access to Teams | Office 365, TeamsAdminAction | No, CloudSIEM Signal created for enablement of external Teams guest access |
VM Run Command | Browse to Azure Portal, Select test Virtual Machine, Navigate to “Run command”, pick any and execute a benign command | Azure Monitor Diagnostic Setting - Administrative Events specifically | No, data source missing, was added as part of exercise |
Delete Virtual machine | Browse to Azure Portal, select test virtual machine, execute deletion | Azure administrative logs | Yes, signal generated |
The team is excited to continue their purple team, and shifts gears to focus on their Azure Kubernetes Service (AKS) deployment.
They would like to know if they have the relevant telemetry and detection logic necessary to ascertain when an external IP address attempts to connect to their cluster.
The team proceeds to execute a simple curl command to attempt to connect to their cluster:
As a next step, the telemetry needs to be examined to ascertain whether this activity is visible.
Rather than specifically searching for a curl user agent, the team wants to start with a broader query that looks at unauthorized requests from external IP addresses to their AKS cluster.
The following query is crafted:
_source="Azure AKS" and _collector="Azure AKS" | where category = "kube-audit" | json field=%"properties.log" "userAgent" as userAgent | json field=%"properties.log" "responseStatus.message" as responseMessage | json field=%"properties.log" "sourceIPs[0]" as sourceIP | json field=%"properties.log" "requestURI" as requestURI | isPublicIP(sourceIP) as is_public_ip | where responseMessage = "Unauthorized" | where is_public_ip | values(userAgent) as user_agents,values(requestURI) as URIs by sourceIP
Everyone is extremely surprised by the results!
Not only is the curl request visible, resulting in an unauthorized response, but many other exploit and discovery attempts from all kinds of IP addresses are seen, including Log4j and Log4Shell exploit attempts:
In this instance and due to the sheer amount of noise hitting the cluster, the team agrees that crafting detections for this type of activity might not be the best use of their precious resources.
The Kubernetes administrator is brought in and shown what kind of traffic is hitting the cluster.
After speaking with management and showing them the results of this purple team execution, everyone is in agreement that this configuration - a publicly exposed AKS cluster - brings unnecessary risk to the enterprise.
A path forward is agreed upon to add an IP allow-list to the cluster, so that not every public IP can attempt to connect to it. A huge win for the purple team exercise!
Many exposed Kubernetes cluster attacks end with the deployment of some kind of crypto miner. It is decided that this activity should be executed and replicated within the AKS cluster.
A basic Ubuntu pod YAML definition is crafted and deployed to the cluster:
A bash shell is executed within the pod, and an Xmrig miner is installed and ran:
After a few minutes of letting the pod execute the Xmrig process, the team receives the following alert:
Because the Sumo Logic Kubernetes solution was deployed on the cluster, deep insights and visibility were available without any additional engineering lift or toil.
In addition to monitoring for high CPU usage, it was also decided to craft a query that looks for pods being created in the cluster.
This would be suspicious activity for this particular Kubernetes cluster, as typically the engineering team responsible for Kubernetes deployments does not deploy pods individually, but rather wraps them in deployment definition.
Indeed, as mentioned previously, collaboration is critical to a purple team exercise, and the FakeORG threat detection engineers were able to learn something new about another team’s internal process, which in turn aided them in threat detection vectors.
The following query is crafted to determine when pods are created:
_sourceCategory=AKS AND _source = "Azure AKS" ubuntu | json field=%"properties.log" "verb" as verb | json field=%"properties.log" "objectRef.resource" as resource | json field=%"properties.log" "requestObject.kind" as request_object | json field=%"properties.log" "requestObject.metadata.name" as object_name | json field=%"properties.log" "sourceIPs[0]" as sourceIP | where verb = "create" | where resource = "pods" | where request_object = "Pod" | values(object_name) as object_name,values(verb) as verb by sourceIP
And the results are returned:
Another alert is crafted for when a terminal is attached to a pod within the cluster:
_sourceCategory=AKS AND _source = "Azure AKS" | json field=%"properties.log" "requestURI" as requestURI | json field=%"properties.log" "sourceIPs[0]" as sourceIP | where requestURI matches /(tty=true)/ | values(requestURI) by sourceIP
And the results:
Once again, the notes are updated:
TTP | Replication Steps | Required Data Source | Existing Detection? |
Password Spray Entra ID | Password spray via CredMaster | Azure AD / Entra ID Sign In Logs | No |
Password Spray Entra ID - 2nd Attempt | Password spray via Credmaster | Azure AD / Entra ID Sign In Logs | Detection crafted |
Register MFA Device | Browse to aka.ms/mfasetup, log in with the relevant user account and follow prompts to register a new multi-factor authentication device. | Azure AD / Entra ID Audit Logs, UserManagement Category | Real-time detection deemed unnecessary, search created and saved |
SharePoint Exfiltration | Create test SharePoint site, run SnaffPoint to search for a sensitive file and then download the file | Office 365 logs, SharePoint operations | Detection Crafted |
Teams External Access Enabled | Browse to Office admin portal, enable external access to Teams | Office 365, TeamsAdminAction | No, CloudSIEM Signal created for enablement of external Teams guest access |
VM Run Command | Browse to Azure Portal, Select test Virtual Machine, Navigate to “Run command”, pick any and execute a benign command | Azure Monitor Diagnostic Setting - Administrative Events specifically | No, data source missing, was added as part of exercise |
Delete Virtual machine | Browse to Azure Portal, select test virtual machine, execute deletion | Azure administrative logs | Yes, signal generated |
Internet Accessible Cluster | Attempt to use curl to connect to clusters external IP address | AKS Diagnostic Logs Enabled - Kubernetes API Server | NA - configuration change made |
CryptoMining | Spin up pod, exec into pod, install Xmrig on pod | AKS Diagnostic Logs Enabled - Kubernetes API Server | Existing metrics-based detection, search looking for pod creations created, search for execution into pod also created |
FakeORG is a relatively new user of Azure DevOps and is just getting started utilizing this service.
A risk for their organization is a threat actor making their way into the DevOps environment and infecting their development pipeline in order to set the stage for a supply-chain style attack.
A requirement is set to monitor all policy changes in Azure DevOps, so the team navigates to the organizational settings menu in Azure DevOps and proceeds to enable Third-party application access via OAuth in order to see if this change can be observed in the telemetry.
The telemetry is examined and a query is crafted:
_source="AzureDevOps" and _collector="AzureDevOps" | %"data.data.policyname" as policy_name | %"data.data.policyvalue" as policy_value | %"data.actorupn" as upn | %"data.ipaddress" as ip_address | where !isNull(policy_value) | where policy_name = "Policy.DisallowOAuthAuthentication" | values(policy_name) as policy_name,values(policy_value) as policy_value by upn,ip_address
And the results returned:
For the next and final execution, the team wants to see if they can capture Azure DevOps permission changes.
A group named “TestGroup1234” is created and the group is granted “Administer workspaces” permissions:
After digging through the telemetry, the following query is crafted:
_source="AzureDevOps" and _collector="AzureDevOps" | where %"data.actionid" = "Security.ModifyPermission" | %"data.data.permissionmodifiedto" as permission_modified_to | %"data.data.changedpermission" as permission_changed | %"data.data.subjectdisplayname" as subject_of_change | %"data.ipaddress" as src_ip | %"data.actorupn" as upn | values(subject_of_change) as subject_of_change,values(permission_changed) as permission_changed,values(permission_modified_to) as permission_modified_to by upn,src_ip
And the results:
And one final time, the notes are updated:
TTP | Replication Steps | Required Data Source | Existing Detection? |
Password Spray Entra ID | Password spray via CredMaster | Azure AD / Entra ID Sign In Logs | No |
Password Spray Entra ID - 2nd Attempt | Password spray via Credmaster | Azure AD / Entra ID Sign In Logs | Detection crafted |
Register MFA Device | Browse to aka.ms/mfasetup, log in with the relevant user account and follow prompts to register a new multi factor authentication device. | Azure AD / Entra ID Audit Logs, UserManagement Category | Real-time detection deemed unnecessary, search created and saved |
SharePoint Exfiltration | Create test SharePoint site, run SnaffPoint to search for a sensitive file and then download the file | Office 365 logs, SharePoint operations | Detection Crafted |
Teams External Access Enabled | Browse to Office admin portal, enable external access to Teams | Office 365, TeamsAdminAction | No, CloudSIEM Signal created for enablement of external Teams guest access |
VM Run Command | Browse to Azure Portal, Select test Virtual Machine, Navigate to “Run command”, pick any and execute a benign command | Azure Monitor Diagnostic Setting - Administrative Events specifically | No, data source missing, was added as part of exercise |
Delete Virtual machine | Browse to Azure Portal, select test virtual machine, execute deletion | Azure administrative logs | Yes, signal generated |
Internet Accessible Cluster | Attempt to use curl to connect to clusters external IP address | AKS Diagnostic Logs Enabled - Kubernetes API Server | NA - configuration change made |
CryptoMining | Spin up pod, exec into pod, install Xmrig on pod | AKS Diagnostic Logs Enabled - Kubernetes API Server | Existing metrics-based detection, search looking for pod creations created, search for execution into pod also created |
Allow third-party OAuth to Azure DevOps | Browse to AzureDevOps portal - Organizational Settings - Policies - Turn on Third-party access via OAuth | AzureDevOps Audit Stream, sending to Event Grid | No, detection crafted |
Grant group “Administer the workplace” permissions in AzureDevOps | Browse to AzureDevOps portal - Organizational Settings - Permissions - Select test group - add necessary permission | AzureDevOps Audit Stream, sending to Event Grid | No, detection crafted |
The team has had a long few days, knee deep in TTPs and queries, and are very keen to wrap up the purple team exercise, debrief and transfer their rough notes to their instance of Vectr to present leadership the results of the purple team exercise.
Once all the relevant information has been entered, the overall purple team looks like:
With the escalation path:
In total, this purple team consisted of:
A MITRE ATT&CK heatmap is also generated for the exercise:
Both the FakeORG team and their leadership is extremely pleased with the results of this purple team exercise.
The team was able to identify new telemetry sources, craft new searches and alerts and perhaps most importantly, collaborate across various functions to better understand use cases and workflows, which resulted in the hardening of their Kubernetes cluster.
The teams’ Sumo Logic Cloud SIEM instance proved to be a critical resource for the duration of the exercise and could handle telemetry at scale from various sources.
The team feels much more confident in their threat detection posture and is already looking to perform this same type of exercise on their Amazon Web Services environment!
Learn more about how a Cloud SIEM solution can help your organization level up security.
Reduce downtime and move from reactive to proactive monitoring.
Build, run, and secure modern applications and cloud infrastructures.
Start free trial