Move and Countermove: Game Theory Aspects of Detection Engineering
Now that I’ve hopefully gotten your attention… I’m also writing a book!
It’s focused more about the philosophical aspects of detection engineering and broader strategic implications. This post is an edited excerpt from the book — which is currently undergoing professional editing before the final release. I’ll post about it once it’s out — I think it’s quite good!
===============================================================
Your PowerShell detection stopped working. Not because it broke — the logic is still executing perfectly. But over the past three weeks, you’ve watched alert volume steadily decline from twenty per week to five per week to zero. PowerShell is still being executed across your environment at normal volumes. You can see it in the raw logs. But it’s no longer triggering your detection.
Then you realize what you’re actually seeing: the threat landscape evolved while you weren’t watching.
Three months ago, a prominent red team published a blog post detailing common PowerShell detection patterns and how to evade them. The post went viral in attacker communities. Now adversaries planning attacks against environments like yours simply avoid those documented patterns from the start. They didn’t need to probe your specific defenses — they studied what everyone’s defenses look like and built evasion into their initial approach. Your detection was effective against last year’s techniques. Today’s attackers learned from public knowledge about detection engineering and simply started somewhere else.
Or maybe it’s simpler: your security vendor pushed an update to their recommended PowerShell detection signatures. Attackers reverse-engineered those signatures, tested their tools against them in lab environments, and adapted their obfuscation techniques accordingly. By the time they target your environment, they’re already using methods designed to evade the exact detection logic you deployed.
This is detection engineering’s uncomfortable truth: you’re not building static defenses against fixed attack patterns. You’re playing a dynamic adversarial game where both sides continuously adapt to each other’s moves. Attackers can study public research about detection techniques, test their tools against common security products, and evolve their tradecraft based on what the defensive community publishes. You observe new attack techniques in breach reports and threat intelligence, update your detections accordingly, and close known gaps. They adapt to the next generation of published defenses. The cycle continues.
Move and countermove. Adaptation and counter-adaptation. An infinite game with no final state.
This circular dependency is exactly what game theory was invented to analyze: situations where rational actors make strategic decisions based on their predictions of what other rational actors will do.
The Evolution of PowerShell Obfuscation: A Case Study in Move and Countermove
Let me show you how this plays out in practice.
In the early days of PowerShell-based attacks, adversaries used relatively straightforward techniques. They’d invoke PowerShell with encoded commands using the -EncodedCommand parameter, which Base64-encodes the command to evade simple string matching. This was effective against defenders who were just alerting on keywords like “mimikatz” or “Invoke-WebRequest” in plain text.
Defender countermove: Detection engineers responded by decoding Base64 command-line arguments and analyzing the decoded content before detection, or by alerting on unusual patterns of Base64 encoding in PowerShell commands. Simple adaptation: if attackers encode their commands, we’ll just decode them first.
Attacker countermove: They started using multiple layers of encoding — Base64 encoding of compressed data, encoding with character substitution, leveraging string concatenation and environment variable expansion to break up detectable strings.
Defender countermove: Detection engineers implemented script block logging, which captures the actual PowerShell code being executed, not just the command-line arguments. Even if the command line is obfuscated, script block logging reveals the deobfuscated commands.
Attacker countermove: Attackers used techniques like AMSI bypass to disable PowerShell’s security features. Still others moved to alternative execution methods — Python, compiled .NET binaries, WMI — techniques that achieve similar objectives without triggering PowerShell-based detections.
Defender countermove: Enhanced detection across multiple execution methods, behavioral analytics that detect the effect of code execution rather than specific tools…
And the game continues.
This progression isn’t just technical evolution — it’s strategic adaptation. Each player’s move is a response to the previous move, and each player must anticipate future moves when deciding current strategy. The attacker encoding PowerShell commands is responding to your string matching. You implementing decode-before-detection is responding to their encoding. They implementing multi-layer obfuscation is responding to your decoding. You implementing script block logging is anticipating their continued use of PowerShell regardless of obfuscation complexity.
This is move-countermove thinking, and it’s fundamentally different from building isolated technical solutions to isolated technical problems.
Why “Living Off the Land” Is Really a Philosophical Statement
For the past years, security researchers began documenting a trend: sophisticated adversaries were increasingly using built-in Windows tools rather than custom malware. PowerShell for execution. WMI for lateral movement. PsExec for remote access. Scheduled tasks for persistence. Native Windows utilities for reconnaissance.
This wasn’t because attackers lacked the capability to write custom malware. Many of these same actors had previously deployed sophisticated, custom-developed tools. They were making a deliberate strategic choice: use the victim’s own infrastructure against them.
This technique became known as “living off the land,” and from a pure detection perspective, it creates profound challenges. The tools themselves are legitimate. PowerShell is installed on every Windows system by design. WMI is fundamental to Windows management. These aren’t malware — they’re normal, expected, necessary system components.
But living off the land is more than just a detection evasion technique. It’s a philosophical statement about the nature of security asymmetry:
Your own infrastructure is your vulnerability. The tools you use to manage your systems are the same tools I’ll use to compromise them. Your legitimate administrative capabilities are my attack vectors. You can’t remove these tools without breaking your own ability to operate.
This creates an existential tension for defenders. The same capabilities that make systems manageable make them exploitable. PowerShell enables automation and management at scale — and enables attackers to execute malicious code. Remote administration tools allow IT teams to manage distributed infrastructure — and allow attackers to move laterally. Every administrative capability is a potential attack vector.
The fundamental asymmetry here is that defenders must balance functionality against security, while attackers simply exploit functionality without concern for operational requirements.
You can’t disable PowerShell across your entire environment because IT operations depends on it. You can’t remove all remote administration capabilities because systems need to be managed. The operational requirements that shape your infrastructure create the attack surface that adversaries exploit.
Let’s get back to PowerShell. You implement measures to detect obfuscated PowerShell. Attackers respond by using PowerShell features in ways that are functionally malicious but structurally legitimate — loading .NET assemblies directly into memory, using native cmdlets in novel combinations, leveraging legitimate PowerShell modules for malicious purposes.
At each step, you’re trying to distinguish between legitimate administrative use of PowerShell and malicious attacker use of PowerShell. But this distinction is fundamentally context-dependent, not technically determinable. The same PowerShell command that an administrator uses to query Active Directory during troubleshooting is the same command an attacker can use during reconnaissance.
The technical artifacts are identical. Only the intent differs.
This is the deep problem of living off the land: it collapses the distinction between legitimate and malicious activity at the technical level. You’re forced to make judgments based on context — who is executing the command, when, from where, following what prior activity, toward what apparent objective. These contextual judgments are inherently uncertain and resource-intensive.
The asymmetry is that attackers can leverage ambiguity while defenders must resolve it.
The Economics of Detection vs. Evasion
Game theory is intimately connected with economics — both deal with rational actors making choices under constraints. In detection engineering, the relevant constraints are time, effort, expertise, and risk. Both attackers and defenders operate under resource limitations, and understanding these economic factors shapes strategic decision-making.
Consider the cost structure of a sophisticated attack:
- Conduct reconnaissance
- Develop or acquire appropriate tools
- Establish initial access
- Maintain persistence
- Escalate privileges
- Move laterally
- Accomplish mission objectives
- Cover tracks
Each phase requires time and expertise. Novel techniques require development effort. Careful operational security requires discipline and slows progress. The more sophisticated the attack, the higher the cost — not necessarily in money, but in time, skill, and effort.
Now consider the cost structure of detection:
- Deploy instrumentation
- Build and maintain collection infrastructure
- Develop detection logic
- Tune detections to minimize false positives
- Investigate alerts
- Respond to confirmed incidents
- Update detections as attacks evolve
Each of these also requires resources. Comprehensive instrumentation consumes storage and processing capacity. Detection development requires expertise. Alert investigation consumes analyst time. The more sophisticated your detection program, the higher the cost.
This creates an economic game where both sides are trying to maximize outcomes while minimizing costs. Attackers want to accomplish objectives with minimum effort and risk. Defenders want to prevent or detect attacks with minimum resource expenditure.
For attackers, this often means exploiting the weakest link rather than attacking strong defenses head-on. If you have comprehensive PowerShell detection but weak authentication monitoring, rational attackers will focus on credential compromise rather than PowerShell-based attacks. If your email security is strong but your VPN has weak multi-factor authentication, attackers will target the VPN. They’re following the path of least resistance — choosing techniques where their effort-to-success ratio is most favorable.
This creates a defensive imperative: avoid obvious weak spots that attackers can exploit cheaply.
If you have world-class endpoint detection but no network monitoring, attackers can move laterally via network protocols without triggering alerts. The economic logic pushes defenders toward breadth of coverage — making all potential attack paths sufficiently costly that attackers don’t have an easy option.
But breadth has its own costs, and resources are finite. This creates an optimization problem: given limited resources, where do you invest in detection to maximize security outcomes?
Game theory suggests focusing on high-leverage detections — those that:
- Cover multiple techniques
- Are difficult for attackers to evade without significant effort
- Protect high-value assets
- Provide detection across multiple phases of attack lifecycle
For example, detecting credential dumping (accessing LSASS memory) is high-leverage because:
- It’s required for many post-exploitation techniques
- It’s difficult to accomplish without generating observable evidence
- It protects authentication credentials that could be used across your environment
Conversely, detecting a specific malware variant is low-leverage:
- It only covers one tool
- Attackers can easily swap tools
- The detection becomes obsolete as the malware evolves
The economics also explain why living-off-the-land techniques are so popular with attackers. Using built-in Windows tools rather than custom malware has economic advantages:
- No development cost
- No need to evade AV/EDR (the tools are legitimate)
- Better operational security (standard tools blend into normal activity)
- Broad applicability (the tools exist in nearly every Windows environment)
For defenders, this is economically challenging because detecting malicious use of legitimate tools requires behavioral analytics and contextual understanding, which is more expensive than signature-based detection of known malware.
Nash Equilibria: Why Some Bad Patterns Persist
In game theory, a Nash equilibrium is a state where no player can improve their outcome by unilaterally changing their strategy, given the other players’ strategies. It’s a stable point — not necessarily optimal for any player, but stable because no one benefits from deviating alone.
Detection engineering has several Nash equilibria that explain why certain patterns persist even when they’re suboptimal for everyone involved.
The False Positive Equilibrium
Defenders would prefer zero false positives — they waste time, create alert fatigue, and train analysts to discount alerts. Attackers would prefer their activities never trigger alerts. Yet false positives persist in a kind of equilibrium.
Why? Because reducing false positives typically requires making detections more specific, which means narrower coverage. If defenders tune too aggressively to eliminate false positives, they create gaps that attackers can exploit. Rational defenders accept some false positive rate because the alternative — missing true positives — is worse.
Simultaneously, attackers can’t completely avoid triggering detections without severely limiting their capabilities. Operating entirely in detection blind spots might mean using inefficient techniques, taking much longer to accomplish objectives, or accepting operational constraints that make success less likely.
The equilibrium state is: defenders maintain detections with non-zero false positive rates because broader coverage is worth the operational cost, and attackers operate in ways that sometimes trigger those detections because completely avoiding detection is too constraining. Neither side is fully satisfied, but neither can improve their position by unilaterally changing strategy.
The Sophistication Equilibrium
Highly sophisticated attacks (custom zero-days, novel techniques, meticulous operational security) are expensive and slow. Unsophisticated attacks (known exploits, commodity malware) are cheap and fast but easily detected. The equilibrium exists in the middle: attackers use moderately sophisticated techniques (living-off-the-land, legitimate tool abuse, basic obfuscation) that are efficient enough to scale but sophisticated enough to evade basic detections.
This equilibrium is reinforced by economic factors. Developing highly sophisticated techniques is expensive, so only high-value targets justify the investment. Most organizations aren’t high-value targets for nation-state actors, so they face attackers using moderately sophisticated techniques. This shapes defensive investment — organizations build detection capabilities appropriate for the threats they actually face.
The result is an equilibrium where attack sophistication and defense sophistication co-evolve at similar levels for any given environment. When defenders broadly adopt a new capability (like widespread EDR deployment), the equilibrium shifts — techniques that were moderately sophisticated become easily detected, forcing attackers to increase sophistication to remain effective.
You Can’t Predict the Unpredictable (But You Can Prepare for It)
Here’s the paradox: you’re trying to predict attacker behavior to build effective detections, but sophisticated attackers deliberately try to be unpredictable. If you could perfectly predict attack patterns, attackers would change patterns to evade prediction.
Game theory offers insight through mixed strategies. In some games, the optimal strategy isn’t choosing a single best move, but randomizing among multiple moves. In rock-paper-scissors, optimal play is choosing randomly with equal probability. Any predictable pattern allows opponents to exploit it.
Attackers use mixed strategies when they randomize techniques, tools, and timing to avoid establishing patterns. Instead of always using PowerShell, they alternate between multiple vectors. Instead of always attacking at night, they vary timing. They introduce random variations that make pattern-based detection more difficult.
But true randomness is actually suboptimal for attackers. Completely random behavior means sometimes choosing inefficient techniques, sometimes exposing themselves unnecessarily. Perfect unpredictability sacrifices effectiveness.
The optimal strategy for attackers is being unpredictable enough to avoid pattern-based detection while still following principles that make attacks successful. And this creates an opening for defenders.
Even when attackers try to be unpredictable, they’re constrained by what works. They need to access specific systems, exfiltrate particular data, accomplish defined objectives. These constraints create necessary patterns — not in specific techniques, but in higher-level behaviors required to succeed.
For example, lateral movement is a necessary behavior for most attacks. The specific technique might vary (PsExec, WMI, RDP, PowerShell remoting), but the fundamental behavior — authenticating to new systems, executing code remotely, establishing sessions on multiple hosts — must occur.
Detection logic that looks for the behavioral pattern “one host initiating connections to many other hosts in a short time window” can catch lateral movement regardless of specific technique. The attacker can’t randomize away the fundamental behavior without abandoning the attack objective.
This is the principle of behavioral detection: focus on what attackers must do, not how they do it.
Attackers can be unpredictable about implementation but can’t be unpredictable about objectives. They must:
- Conduct reconnaissance (observable as unusual enumeration)
- Establish persistence (observable as new scheduled tasks, services, registry modifications)
- Escalate privileges (observable as credential access or exploitation)
- Accomplish objectives (observable as data access, exfiltration, or system manipulation)
The sequence might vary, timing might vary, specific techniques might vary, but the categories of behavior are predictable because they’re necessary.
For detection engineers, this means: instead of trying to predict “this attacker will use technique X at time Y,” ensure that whichever technique they choose from the set of viable techniques, you have some probability of detection.
Think of it as portfolio theory in finance. You don’t try to predict which specific stock will perform best. You build a diversified portfolio that performs reasonably well across various conditions. Similarly, you don’t try to predict which specific technique an attacker will use. You build a diversified detection portfolio covering multiple attack vectors, techniques, and behaviors, ensuring most attacks trigger at least some detections.
Thinking Multiple Moves Ahead
When designing a detection, experienced engineers ask not just “will this detect the current technique” but “how will attackers adapt when they encounter this detection, and will their adaptations still be detectable?”
If your detection for PsExec causes attackers to switch to WMI, and you have no WMI detection, you’ve just pushed them into a blind spot. Better to deploy WMI detection first, so that when attackers adapt away from PsExec, they adapt into another monitored technique.
This strategic thinking changes everything:
Instead of asking: “How do I detect this malware?”
Ask: “How do I make this entire class of techniques more costly for adversaries?”
Instead of asking: “How do I reduce false positives?”
Ask: “What trade-offs am I making between coverage and noise, and are those trade-offs appropriate for my threat model?”
Instead of asking: “How do I predict what attackers will do next?”
Ask: “How do I ensure that viable attack options all have detection coverage, regardless of which specific option adversaries choose?”
This is the shift from tactical to strategic thinking. From reactive to proactive. From technical problem-solving to adversarial reasoning.
The Takeaway: You’re Playing Whether You Know It or Not
Detection engineering is a strategic game whether you recognize it as one or not. Every detection you build is a move. Every attacker adaptation is a countermove. The game continues with or without your conscious participation in its strategic dimension.
The question isn’t whether to play strategically — you’re already playing. The question is whether you’ll play consciously or unconsciously. Whether you’ll think about move-countermove dynamics or just react to immediate technical problems. Whether you’ll shape the game deliberately or let it shape you.
Understanding game theory doesn’t make detection perfect. The asymmetries remain. The adversary still has advantages. You still can’t detect everything. But it changes how you think about the problem space, how you design detections, how you invest resources, how you measure success.
You stop building isolated technical solutions and start building strategic capabilities. You stop thinking about detection as a series of independent challenges and start thinking about it as positions in an ongoing game. You stop optimizing for immediate threat detection and start optimizing for sustained defensive capability over time.
The game never ends. But understanding that you’re playing it — really understanding the strategic dynamics — makes you a more effective player.
And in an infinite game, effectiveness over time is the only victory that matters.
If you enjoyed the article, feel free to connect with me!
https://www.linkedin.com/in/daniel-koifman-61072218b/
https://x.com/KoifSec
https://koifsec.me
https://bsky.app/profile/koifsec.bsky.social
Move and Countermove: Game Theory Aspects of Detection Engineering was originally published in Detect FYI on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.
1 post - 1 participant
Malware Analysis, News and Indicators - Latest topics
Sp123
"An underestimated security threat to organizations is employee apathy and burn out."