An Analyst’s Dilemmas
This article originally appeared on AttackIQ Informed Defenders with the title "An Analyst's Dilemmas"
Author Sai Molige
Have you ever watched the clip where Charlie Chaplin works in a factory? If you have not, then give it a look: it is called “Modern Times-Factory Work.” To summarize, Chaplin has to work on a metal piece very fast, without missing a single item, before it reaches another person for the next phase, and the chain continues until the product reaches more machinery. The reason I mention this is that I feel like in some SOC environments, the analysts are like Chaplin, having to work on parts (alerts) FAST, WITHOUT ERROR, and PASS ON TO OTHERS.
We will take a look at some of the issues SOCs face from these moving parts that may prevent an analyst’s efficiency and consistency in their triaging process. More importantly, we will also discuss what an analyst may do to overcome these issues. Remember, many things are above the analyst’s pay grade, but we’ll only discuss what we as analysts have control over.
I have divided this blog into four parts: The Usual, Architectural, Log Augmentation, and Alerts. Later we will discuss some of the things an analyst may do to help themselves.
The number of alerts some SOCs have to deal with is just shy of 25,000/day. Because of this, two issues arise: analyst burnout and tribal knowledge, or unwritten knowledge not commonly known within a team. Due to the increase in the number of alerts, an analyst’s already increased workload just spiked. Now add “less efficient/skilled staff” to the mix. These two factors increased the resources (time, energy, etc.) required from an already drained analyst. Tribal knowledge will not scale well if it is not transferred to the rest of the team. If you do not think it is an actual problem, read the “Phoenix Project” book and you will understand its effect at the team/organizational level.
To better understand how each component hinders an analyst’s work, let’s use an example. An analyst (let’s call the analyst Chaplin) wants to work on the alert, but he does not have the rule the alert triggered on nor does he have all the context required around the triggered alert. So, he took whatever info he had from the alert and searched in the SIEM’s storage using SIEM’s search engine. The info he wants is not populating right away, as there is too much data in SIEM’s storage, and the SIEM is not designed to handle significant amounts of writing, indexing, and querying. So, Chaplin decided to trim his search and decided to only concentrate on a particular data set, and after crawling for a while he saw some results related to the alert. The documents or logs returned are not enriched, have no common naming convention with which he can correlate with other log sources, and, on top of that, the results are skewed because of time format/zone issues.
SIEM is full of logs:
Some of the questions one might ask are: what is the problem, why it is so important, and how does it affect an analyst? Let’s address them one by one:
Due to the lack of understanding of the log on-boarding and enrichment process:
We may have “blind spots” on what we see and protect
Inconsistencies in the field naming conventions
Ugly blocks of data that aren’t searchable
Adverse impact on the query search time and query return time
Missed opportunity on using out-of-box SIEM alerts and ability to create custom alerts
This gives us a nice segue into our last set of problems in a SOC: alerts.
Alerts, where the life of an analyst revolves around:
This is one of the places an analyst spends a lot of time in a SIEM. An analyst is normally either hired, appraised, and even fired on the number of alerts they triage. Like logs, alerts can also be generated from various places and some of the most common ones are Intrusion Detection/Prevention Systems, firewalls, and Anti-Virus (AV) engines. Though you will get some information from the point products, it doesn’t give an analyst the full picture of what happened. Some of the issues an analyst faces with alerts generated by point products are: