Fine-tuning Cloud SIEM detections through machine learning

Security engineering teams spend hours every week tuning their security information and event management (SIEM) systems to ensure that they are effective at detecting security threats and minimizing false positives. Such “tuning tax” is common as customers add new SIEM rules to cope with rapidly changing threat landscape and attacker tactics and as their attack surface evolves through automated changes to their application and infrastructure stacks.

Just as you would engage a personal trainer to get the best results for time spent at the gym, Insight Trainer learns from your Signals and Insights data to identify Tuning Expressions and adjustments to rule severities to optimize your Cloud SIEM detections. With Insight Trainer, SOC teams can free themselves to focus on real threats by minimizing time spent investigating false positive Insights.

A day in the life of a security engineer

Many Sumo Logic customers divide responsibility between security/detection engineers and the Security Operations Center (SOC) teams. The latter are the first-line responders for security incidents while the former administer and tune SIEM platforms like Sumo Logic for detection efficacy.

A typical process for tuning SIEMs involves the following steps:

1. Gather feedback from SOC teams. Many customers have weekly meetings between security engineers and SOC teams to assess the latest security findings including rules that seem to be noisy as they cause false alarms.

2. Refine detection rules. Security/detection engineers analyze Insights and Signals from noisy rules within Cloud SIEM and assess if known and trusted users (e.g. system administrators) or machines (e.g. instances created by trusted automation scripts) dominate false positives.

3. Create Tuning Expressions that mute signals from these users or lower the severity or deactivate these rules. In any of these strategies, the impacted rules are expected to trigger fewer insights because Cloud SIEM’s detection algorithm triggers Insights when the cumulative severity of rules triggered on an entity exceeds a threshold.

4. Rinse and repeat the above process every week.

Security engineers also assess new and emerging threat patterns from penetration testing or other means and add new detection rules, initially as prototype rules. A prototype rule generates Signals, but those Signals will not contribute to Insights. Running the rule as a prototype for a while allows you to determine whether the rule is too noisy and fires too many Signals.

Many inefficiencies are apparent in this process including

• The tuning process is manual. There is only so much analysis security teams can do through log searches that it is unlikely for the tuning process to be comprehensive.

• Finding optimal rule severities involves trial and error. Setting severities too low on high efficacy rules can miss real threats while setting them too high on low efficacy ones can result in too many false alarms.

• Given that security teams focus on false positives exclusively, they do not assess if certain rules are particularly helpful for detecting real threats or true positives. Increasing the severity of such rules can catch more threats sooner, given the way algorithmic insights are generated by Cloud SIEM.

• Determining Tuning Expressions for noisy entities is also manual and unlikely to be comprehensive.

• New detection rules tend to be noisy. While Cloud SIEM’s rule prototyping workflow allows experimentation before activating rules in production, that tuning process still involves trial and error.

• Tuning has to be a continuous process as the threat, application and infrastructure landscape is constantly changing, especially with heavy automation through Infrastructure-as-code. Manual tuning processes can compound the “tuning tax” and consume valuable time from detection/security engineering teams.

Cloud SIEM Insight Trainer is designed to alleviate the manual tuning burden and maximize Cloud SIEM efficacy while still aligning with established security engineering processes as explained below.

Cloud SIEM Insight Trainer

Cloud SIEM Insight Trainer is a dashboard packaged within CSE as shown in the screenshot.

Insight Trainer learns from the history of your Cloud SIEM Insights and associated SOC team resolutions: true positive / resolved, false positive or no action. The algorithm calculates rule severities that accomplish the following goals

• Preserve true positive / resolved counts

• Minimize false positives

• Optionally, minimize no action. This option is appropriate for customers who use false positive and no action resolution states interchangeably.

Both Sumo Logic Threat Labs and custom rules are included in the severity recommendations. While the details of our machine learning algorithm are proprietary, all other things being equal, the algorithm recommends severity increases for rules that consistently contribute to true positive insights. Conversely, rules that consistently contribute to false positive insights are recommended for severity reduction.

In addition to rule severity recommendations, Insight Trainer analyzes the data to automatically identify noisy entities through a Tunability score. High Tunability score rules are those whose false positive (and optionally, no action) insights are caused by a small number of entities. Such rules would benefit from Tuning Expressions to suppress noisy entities, especially, if such entities are verified to be safe. Often, service accounts and administrative users may trigger Insights resolved as false positives or no-action insights by many customers.

To use Insight Trainer, we recommend the following workflow:

1. Review severity recommendations by rule

2. Assess dominant entities in false positives / no actions through rule Tunability scores

3. Evaluate and add tuning expressions for dominant entities, where possible.

4. Adjust rule severities for other rules.

For best results, we also recommend security teams define and adhere to a standardized definition of true positive, false positive and no action resolution states. To avoid ambiguity during investigations, some Sumo Logic customers have used Insight sub-resolution states to guide SOC responders along with upfront training on how to interpret and assign resolution states for common scenarios like admin activity, penetration testing or VPN-related Insight activity.

Insight Trainer in action

We ran Insight Trainer over hundreds of Cloud SIEM customers and observed the following results. The average drop in false positives was 72% while no action insights was 74%. On average, only six or seven rules were recommended for severity decrease while zero or one rule was recommended for severity increase. In other words, tuning a few rules can result in dramatic reductions in False Positive / No Action Insights

Several Sumo Logic customers evaluated Cloud SIEM Insight Trainer over the past two months and were pleased with the results. A detections engineer from a financial services organization reported:

“...looking at these rules [severity recommendations], we recognize many common offenders…changing severity by just 1 can reduce the false positive count [associated with rule] by 60% which is HUGE. Also, with growing AWS adoption alongside automation usage, we expect to see a lot of false positives for new rules.

It is very difficult to tune these out through Tuning Expressions because service accounts used by automation are short-lived but are making lots of changes in a short amount of time - triggering a lot of insights. Such legitimate activity is the greatest contributor of false positives in an AWS environment and severity changes are a key lever to mitigate these.”

ChatGPT envisions Insight Trainer

As a fun exercise, we asked ChatGPT about typical false positive rates for SIEM systems and goals security engineering teams should aspire for, on this measure. Of course, ChatGPT is a limited tool without knowledge of the truth (whatever the truth even is!), but it can reflect fascinating patterns in existing data. Below is ChatGPT’s response.

“Generally, a high false positive rate can be an indicator that the SIEM system is not optimally configured or tuned. There is no one-size-fits-all answer to what a typical false positive rate for SIEMs should be.

However, it is common to aim for a false positive rate of less than 5% to ensure that the security team can focus on genuine security incidents and reduce alert fatigue. It is important to note that reducing the false positive rate should not come at the expense of reducing the true positive rate, which measures how accurately the SIEM system identifies genuine security incidents.

The goal is to strike a balance between reducing false positives and ensuring that the SIEM system can accurately detect and respond to genuine security incidents.”

We could say that Insight Trainer would make ChatGPT happy if it were possible to make an AI language model happy!

Next steps

Cloud SIEM Insight Trainer is an essential tool for fine-tuning Cloud SIEM detections for your organization and helps focus SOC analysts’ attention on high-risk, true positive insights. Start using Insight Trainer today, or learn how Roku tunes its SIEM.