Author: Kostas
Threat hunting is becoming mainstream, and despite the attention it receives, many people need help to differentiate it from other roles, such as detection engineering. This confusion leads to endless discussions on places like Twitter and Reddit.
I wrote this article to share my perspective on what makes threat hunting unique regarding its approach to solving a specific problem. I have a special place in my heart for both of these roles and I believe working together is the best approach no matter the method you choose to operate.
I tried to remain unbiased throughout the article in my effort to highlight both similarities and differences between detection engineering and threat hunting. Although you should consider that this article is part of the threat hunting series, I am a threat hunter, and confirmation bias is hard to eliminate entirely. Any feedback is welcome!
The differences
Threat hunting
Threat hunting is a proactive practice of looking for evidence of adversarial activity that conventional security systems may miss. It entails actively searching for signs of malicious behaviour and abnormalities in a network or individual hosts. Detection engineering, on the other hand, is the process of developing and maintaining detection methods to identify malicious activity after it has become known.
Although detection engineering could also be done proactively by creating detections before a malicious activity occurs, it focuses on reacting to threats that have already been observed. Threat hunting’s approach is to search for threats that are present in the network and may circumvent current detections in one way or another. This includes threat actors’ efforts to remain undetected by employing living-off-the-land tools and other benign methods.
Because of this important distinction, threat hunting takes a different approach when it comes to querying and analyzing the data. Since threat hunting aims to identify threats that might have circumvented detections, threat hunters should understand how the current detections are structured. Depending on the initial hypothesis, threat hunters can then concentrate on techniques that are not covered by current detections or techniques too difficult to detect due to noise(high false positive rates). An alternative approach would be to search for signs of early intrusion based on anomalous behaviour.
An example is searching and analyzing the usage of the least commonly used communication protocols during the past month(maybe we can find RedLine stealer infections using Windows Communication Foundation (WCF) protocol 😉).
Threat hunting queries can have a wider tolerance of noise in the results. Because of that, multiple data-analysis techniques can assist with zeroing into the outliers. I have written about some of these techniques that I find helpful daily here, but there is an excellent article from CyborgSecurity that explains many others in a simplistic manner.
Detection Engineering
Unlike threat hunting, detection engineering has been around for a long time, with the first signs of Intrusion Detection Systems(IDS) traced back to the 80s. Detection engineering concepts have matured and developed with time. Besides automated detections for static atomic indicators, modern detection engineering incorporates indicators based on threat actors’ behaviours and tools. Detection rules undergo multiple phases before they finally end in signals of malicious activity, aka detection alerts. In contrast to threat hunting, these efforts concentrate on known indicators and serve a specific purpose, creating signals that will capture malicious activity.
The detection engineering process is also vastly different from threat hunting. Below are some significant tasks that comprise part of that process.
One of the most important and time-consuming tasks in detection engineering is dealing with false positives. An abundance of false positives can become a concern(ask any infosec analyst 😂), so engineers must walk a fine line between abstract and specific detections. Another important task is analyzing the current state of detections and tuning existing rules based on their performance. Similar to the development process, rules undertake a testing and development period before they make it into production.
Threat hunters don’t have to worry about the detection process. However, they are expected to spend significantly more time analyzing a vast amount of structured and unstructured data for signs of intrusion or corporate policy violations.
The similarities
Despite the different approaches, threat hunting and detection engineering share similarities regarding operational tasks. Even though the scope differs, some querying and analysis steps could overlap. For example:
A threat hunter might search for a specific Indicator Of Attack(IOA) as reported by the threat intelligence team. The search will be broad, and the results might contain false positives. The threat hunter will go through the data and investigate the results that match the expected IOA. If suspicious activity is uncovered, the information will be passed on to the incident response team. Depending on the intrusion phase of this finding, the threat hunter will continue to hunt for earlier signs of intrusion or later evidence of post-exploitation activities. The hunter might also look for other possible ways that the same attack could occur in an environment and even dive into unsuccessful exploitation attempts looking for other evidence of the attack.
A detection engineer might create a similar query to inspect the signal-to-noise ratio before creating the initial testing rule. The search will be specific to the expected IOA. If suspicious activity is uncovered, the information will be passed on to the incident response team.
From the above example, both roles involve analyzing data to identify potential security threats. To do that, they require the same understanding of the environment they are trying to protect. We also see how both functions engage in threat research and consume threat intelligence information to stay on top of the latest threats.
Lastly, both threat hunting and detection engineering entail simulating threats to gain a deep understanding of how to hunt or detect them accordingly. The difference in this task is how the two roles handle the emulation results.
The diagram below contains a high-level overview of the threat hunting and detection engineering tasks; it is only a partial breakdown and does NOT represent all the tasks for each role. I made the graph to demonstrate the differences and similarities between the two jobs for those visual learners out there 🙂.
Conclusion
Threat hunting and detection engineering are often confused because both relate to identifying potential threats or malicious activity on a system or network. As demonstrated in this article, threat hunting and detection engineering have different goals and operational tasks. Because of that, an organization should consider separating these two roles. Having separate teams with dedicated personnel can help ensure that each team focuses on their specific tasks, which could improve the overall efficacy of the security program.
Having said that, having a single team handling both detection engineering and threat hunting could be more efficient and cost-effective for smaller-sized organizations. Unfortunately, in such cases, the people doing both functions and those overseeing the process tend to confuse them as one indistinguishable role. However, they can only experience the full potential of either after separating the duties.
Threat hunting, detection engineering and threat intelligence should work together to form a complete information security program. Every organization has a different security maturity level, affecting how the information security program is structured. Understanding the various functions within information security can help organizations scale their program effectively.
Follow me here and on Twitter for updates on the next posts for this series.