Tuned by Design: Why Detection Engineering Needs Its Own Development Lifecycle
We embraced “Secure by Design” in software development. It is time we applied the same philosophy to SOC detection content — introducing the Use Case Development Lifecycle.
If you have spent more than a year inside a Security Operations Center, you have lived this story: a brand-new analytic rule goes live on Monday morning. By Tuesday afternoon the SOC is drowning in false positives. An analyst mutes the rule. A month later, a real intrusion slips through the exact blind spot that rule was supposed to cover. Post-incident, management asks the inevitable question — “Why didn’t we detect this?”
The answer is painfully simple. The rule was never engineered to survive production. It was written, deployed, and forgotten — a pattern so common that most SOC teams treat it as an unavoidable reality rather than a solvable engineering problem.
In software engineering, we solved a strikingly similar problem years ago. The DevSecOps movement taught us that security cannot be bolted on at the end of a release cycle; it must be woven into every stage of development. That principle — “Secure by Design” — has reshaped how we build applications. But the security operations world, the very domain responsible for detecting threats after deployment, has never adopted an equivalent philosophy for its own detection content.
This is the gap I want to close. I call it Tuned by Design.
“If Secure by Design means building applications that are inherently resistant to attack, then Tuned by Design means building detection rules that are inherently resistant to noise, blind spots, and operational decay.”
Tuned by Design is not a product. It is not a framework you download. It is a philosophy — a commitment that every detection use case should pass through a rigorous, repeatable development lifecycle before it ever touches a production environment. And once it reaches production, it should be continuously measured, maintained, and retired with the same discipline we apply to application code.
Before introducing Tuned by Design, I want to acknowledge Alex Teixeira’s 2017 Opstune article, “Mapping SDLC to security use cases development process,” which made an important early case for applying SDLC thinking to SIEM and security use case development. The article argued that detection logic, dashboards, and other security knowledge objects should be treated like code: defined, developed, tested, improved, and maintained. This post builds on that foundation and extends it into a modern detection engineering lifecycle focused on production tuning, validation, operational metrics, and long-term rule health. Check it out from here -> Mapping SDLC to security use cases development process — Opstune.com
In this article, I will break down why the current approach to detection engineering fails, what Tuned by Design means in practice, and how to implement a full Use Case Development Lifecycle (UCDL) that transforms detection content from disposable queries into durable, high-fidelity security assets.
The Problem: Detection Content Is Treated as Disposable
Walk into most SOC environments and ask a simple question: “How many of your analytic rules have been reviewed in the last 90 days?” The answer is almost always uncomfortable. Detection rules accumulate like sediment. New rules are added after every threat intelligence briefing, every penetration test, every compliance audit — but almost none of them are ever revisited, tuned, or deprecated.
The consequences are predictable and devastating. Alert fatigue crushes analyst morale. True positives are buried under mountains of benign noise. Coverage gaps appear silently because nobody is mapping rules against the threat landscape in a structured way. And when a breach occurs, the post-mortem reveals that the organization had the right data sources and the right detection logic — just never tuned, validated, or maintained.
This is not a tooling problem. Microsoft Sentinel, Splunk, CrowdStrike, Elastic — every modern SIEM and XDR platform provides powerful detection capabilities. The problem is process. Or more accurately, the absence of one.
The Parallel to Pre-DevSecOps Software Development
Consider how software was built before the Secure Development Lifecycle (SDL) and DevSecOps became standard practice. Developers wrote code, QA tested it at the end, and security was an afterthought — a penetration test conducted two weeks before launch. The result was predictable: vulnerabilities in production, expensive patches, and breaches that could have been prevented by a simple threat model early in the design phase.
The SDLC revolution introduced structure. Requirements gathering, threat modeling, secure code review, automated testing, staged deployments, and continuous monitoring became standard phases. Security stopped being a gate at the end of the pipeline and became a thread woven through every stage.
Detection engineering today is where software development was fifteen years ago. We write rules and push them to production with minimal validation, no formal testing methodology, no tuning criteria defined upfront, and no retirement plan. We need our own lifecycle.
The reactive anti-pattern: detection rules go from keyboard to production with no engineering rigor, inevitably resulting in alert fatigue and coverage blind spots.What “Tuned by Design” Actually Means
Tuned by Design is a philosophy that borrows its core principle from the Secure by Design movement: quality must be architected in from the beginning, not patched in after failure.
In practical terms, it means that every detection use case — whether it is a Sentinel analytic rule, a Defender XDR custom detection, a Sigma rule, or a Splunk correlation search — should be treated as a product with its own requirements, design specification, test plan, deployment strategy, and maintenance schedule.
A detection rule that is “Tuned by Design” exhibits five characteristics:
- Threat-Informed. It is mapped to a specific adversary technique (MITRE ATT&CK), a specific threat actor profile, or a concrete business risk — not written in a vacuum.
- Data-Aware. The required data sources, schemas, and log fidelity have been verified before the rule is written. If the telemetry does not exist or is unreliable, the rule does not get built.
- Pre-Tuned. Exclusion logic, baseline thresholds, and known benign patterns are defined during development, not discovered in production through analyst pain.
- Validated. The rule has been tested against both simulated attack data (true positive validation) and representative production data (false positive estimation) before deployment.
- Observable. The rule ships with defined metrics — expected alert volume, expected true positive rate, SLA for triage — and these metrics are monitored continuously.
Key Insight
Tuned by Design does not mean the rule will never produce a false positive. It means that false positive behavior has been anticipated, measured, and accounted for before the rule reaches the analysts who must respond to it. The goal is not perfection — it is predictability.
The Tuned by Design philosophy maps directly onto the Secure by Design model that transformed software development. Each SDLC phase has a detection engineering counterpart.The Use Case Development Lifecycle (UCDL)
If Tuned by Design is the philosophy, then the Use Case Development Lifecycle is its operational implementation. The UCDL is a six-phase framework that governs how detection use cases are conceived, built, deployed, measured, maintained, and eventually retired.
Each phase has defined inputs, outputs, quality gates, and responsible roles. Skipping a phase is possible in emergencies (zero-day response, for example), but the skipped phase must be revisited retroactively. No detection rule should remain in production without having completed every phase.
The complete Use Case Development Lifecycle. Note the continuous feedback loop from production back to ideation — detection engineering is never “done.”Let us walk through each phase in detail.
Phase 1: Ideation — Start with the Threat, Not the Query
Every use case begins with a question: “What adversary behavior are we trying to detect, and why does it matter to this organization?” The ideation phase forces detection engineers to anchor their work in threat context rather than chasing the latest community Sigma rule without understanding whether it applies to the local environment.
During this phase, the team reviews current threat intelligence — both strategic (threat landscape reports, industry-specific campaigns) and tactical (indicators of compromise, adversary playbooks). Each candidate use case is mapped to a MITRE ATT&CK technique and a business risk. A detection for Kerberoasting, for example, is not just “T1558.003” — it is tied to the specific business impact of credential compromise in the organization’s Active Directory environment.
The output of this phase is a Use Case Brief: a one-page document that captures the threat context, the ATT&CK mapping, the expected data sources, and a preliminary priority score based on business risk and threat relevance.
Phase 2: Feasibility — Validate Your Data Before You Write a Single Line
This is the phase that most organizations skip entirely, and it is arguably the most important. Feasibility is where you answer the question: “Do we actually have the telemetry required to detect this technique reliably?”
A detection rule for lateral movement via PsExec, for instance, requires Windows Security Event Logs with process creation auditing enabled (Event IDs 4688 with command-line logging), Sysmon data, or EDR telemetry from Microsoft Defender for Endpoint. If the target environment only forwards authentication events and not process creation, the rule will never fire — or worse, it will fire on partial data and produce unreliable results.
The feasibility assessment checks data source availability, schema consistency across the environment, ingestion latency, retention periods, and any known gaps (specific endpoints not forwarding logs, legacy systems with limited telemetry). If the data does not exist or is unreliable, the use case is either deferred (with a data onboarding request filed) or redesigned to work with available telemetry.
Phase 3: Development — Engineering, Not Scripting
With a validated Use Case Brief and a confirmed data foundation, the detection engineer builds the detection logic. In a Tuned by Design workflow, development is not just writing a KQL or SPL query. It includes three components built in parallel:
The Detection Query. The core analytic logic, written in the appropriate query language for the target platform. This is where the engineer defines what “suspicious” looks like — the signal extraction.
The Exclusion Baseline. Before the rule reaches production, the engineer runs it against historical production data (typically 7–30 days) to identify known benign patterns. Legitimate service accounts, scheduled tasks, IT automation tools — these are documented and encoded as exclusions during development, not after the first wave of false positives in production.
The Response Playbook. Every detection use case ships with a corresponding analyst playbook: what this alert means, what to investigate first, what context to enrich, what escalation criteria apply, and what the expected response SLA is. Detection without response guidance is incomplete engineering.
Phase 4: Validation — Test Like You Mean It
Validation is the quality gate that separates Tuned by Design from the traditional “deploy and pray” approach. This phase has two components:
True Positive Validation. The detection engineer simulates the adversary technique in a test environment (or a controlled segment of production) using tools like Atomic Red Team, Caldera, or manual attack simulation. The goal is to confirm that the rule fires correctly when the technique is executed. If it does not, the logic is revised.
False Positive Estimation. The rule is run in a monitoring-only mode (no alerts generated) against live production data for a defined observation period — typically 5 to 14 days. The engineer reviews every match, classifies it as true positive, benign true positive, or false positive, and calculates an estimated false positive rate. If the rate exceeds the team’s threshold (many mature SOCs target below 20%), the rule goes back to development for additional tuning.
Both validation activities are documented, and the results form part of the use case’s permanent record. A peer review gate ensures that at least one other engineer or analyst reviews the logic, the exclusions, and the validation results before the rule is promoted to production.
The validation gate requires both true positive confirmation and false positive estimation before any rule reaches production. Failed rules return to development for tuning.Phase 5: Production — Deploy, Measure, Tune Continuously
A rule that passes validation enters production — but production is not the finish line. It is the beginning of a continuous measurement cycle. Every rule in production should be tracked against a set of defined KPIs:
- Alert Volume: Is the rule generating alerts within the expected range? A sudden spike or drop is a signal that something has changed — in the environment, in the data, or in the threat landscape.
- True Positive Rate: What percentage of alerts from this rule result in confirmed security incidents? This is the single most important metric for detection quality.
- Mean Time to Triage (MTTT): How long does it take an analyst to investigate and classify an alert from this rule? Long triage times may indicate that the alert lacks context, the playbook is insufficient, or the rule is too ambiguous.
- Coverage Contribution: Which ATT&CK techniques does this rule cover, and how does it contribute to the organization’s overall detection coverage map?
Rules that consistently underperform — high false positive rates, low true positive rates, or excessive triage times — are flagged for a tuning cycle. Tuning is not a one-time event; it is a recurring maintenance activity, ideally on a quarterly cadence for all production rules.
Phase 6: Retirement — Detection Rules Have an Expiration Date
This is the phase that almost nobody implements, and its absence is one of the primary drivers of rule bloat and SOC entropy. Detection rules are not permanent. Threat techniques evolve. Infrastructure changes. Data sources are decommissioned. A rule that was critical eighteen months ago may now be irrelevant, redundant, or actively harmful (generating noise that distracts from current threats).
The retirement phase introduces a formal deprecation review. Rules are evaluated against current threat intelligence, current infrastructure coverage, and current performance data. Rules that no longer serve a purpose are disabled, documented, and archived — not simply left running because nobody remembers who created them or whether they are still needed.
The Business Case: Why Leadership Should Care
For security leaders and CISOs reading this, the natural question is: “This sounds like more overhead. Why should I invest in a detection development lifecycle when my team is already stretched thin?”
The answer is in the numbers. Consider the total cost of a poorly engineered detection rule:
Direct analyst cost. A single noisy rule generating 50 false positive alerts per day, each requiring 10 minutes of triage, consumes over 8 hours of analyst time daily — the equivalent of a full-time employee doing nothing but chasing ghosts. Across 20 noisy rules, you have consumed your entire L1 analyst team’s capacity on non-productive work.
Opportunity cost. Every minute spent triaging a false positive is a minute not spent investigating genuine threats, hunting proactively, or improving security posture. Alert fatigue does not just waste time — it creates blind spots.
Incident cost. When a real intrusion is missed because the relevant alert was muted or buried in noise, the cost is measured in breach impact: regulatory fines, remediation expenses, reputational damage, and business disruption. The average cost of a data breach continues to rise annually, and “we had a rule but it was too noisy” is an increasingly common finding in post-incident reviews.
Tuned by Design reduces all three cost categories. By investing a modest amount of engineering effort upfront (feasibility validation, pre-tuning, simulation testing), organizations dramatically reduce the ongoing operational cost of their detection program and significantly improve their probability of catching real threats.
Tuned by Design requires a slightly higher upfront engineering investment but dramatically reduces the compounding costs of false positives, analyst burnout, and missed detections over time.Implementing Tuned by Design: A Practical Roadmap
Adopting a detection development lifecycle does not require a massive transformation program. It requires three things: a defined process, a lightweight governance structure, and a commitment to measurement. Here is a practical roadmap for teams ready to move from reactive detection management to Tuned by Design engineering.
Step 1: Establish a Use Case Backlog
Create a centralized backlog of all detection use cases — both existing rules and proposed new detections. Each use case should be tracked with its current lifecycle phase, its ATT&CK mapping, its data dependencies, and its performance metrics (once in production). This can be as simple as a structured spreadsheet or as sophisticated as a dedicated platform, but it must exist as a single source of truth.
Step 2: Define Your Quality Gates
Agree on the criteria that must be met before a rule advances from one phase to the next. At minimum, define thresholds for data availability (Phase 2 gate), true positive validation (Phase 4 gate), and false positive rate (Phase 4 gate). Document these thresholds and make them visible to the entire team.
Step 3: Introduce Detection-as-Code
Version-control your detection content. Store analytic rules, exclusion lists, and response playbooks in a Git repository. Use pull requests for peer review. Implement CI/CD pipelines that automatically validate query syntax, check for schema dependencies, and run rules against test data before deployment. This is not optional — it is the single most impactful practice for operationalizing the UCDL.
Step 4: Build a Detection Coverage Map
Overlay your detection use cases onto a MITRE ATT&CK matrix to visualize where you have coverage, where you have gaps, and where your coverage is concentrated. This map becomes the strategic planning tool for prioritizing new use case development and identifying over-invested areas that may benefit from rule consolidation.
Step 5: Institute a Quarterly Review Cycle
Every quarter, review all production rules against their defined KPIs. Flag underperforming rules for tuning, identify rules that should be retired, and reprioritize the development backlog based on current threat intelligence. This review is the heartbeat of the continuous feedback loop that keeps the program aligned with reality.
The Tuned by Design maturity model. Most organizations sit at Level 1. Moving to Level 3 — the “Tuned by Design” baseline — requires process, not products.Technical Deep Dive: What a “Tuned by Design” Use Case Looks Like
To make this concrete, let us walk through a single use case — detecting Kerberoasting activity in a Microsoft Sentinel environment — and show what it looks like when developed through the UCDL.
Ideation Output
Technique: Kerberoasting (MITRE ATT&CK T1558.003). Threat context: adversaries request Kerberos service tickets for service accounts with SPNs, then crack the tickets offline to obtain plaintext credentials. Business impact: compromised service accounts often have elevated privileges in Active Directory, enabling domain-wide lateral movement. Priority: High — recent threat intelligence indicates active use of this technique by multiple threat actors targeting our industry vertical.
Feasibility Assessment
Required data: Windows Security Event ID 4769 (Kerberos Service Ticket Operations) with Ticket Encryption Type and Service Name fields populated. Validation: confirmed ingestion from all domain controllers into the SecurityEvent table in Sentinel. Average ingestion latency: 2–4 minutes. Retention: 90 days in the hot tier, 365 days in the Data Lake (basic logs). Schema consistency: confirmed across all three domain controllers. Verdict: Feasible.
Development Artifacts
The detection query targets Event ID 4769 where the Ticket Encryption Type uses RC4 (0x17) — the weaker encryption type that Kerberoasting tools request by default — while excluding machine accounts (which end with $), known service accounts that legitimately use RC4, and health-check or monitoring accounts. The exclusion baseline was derived from a 14-day analysis of production data that identified three service accounts and one monitoring tool consistently triggering this pattern. The response playbook instructs analysts to identify the requesting account, correlate with recent logon activity, check for abnormal SPN enumeration (Event ID 4769 spikes), and escalate to Tier 2 if the requesting account is not on the known-benign list.
Validation Results
True positive test: Rubeus Kerberoasting executed against a test SPN in the lab environment. Alert fired within 3 minutes. Confirmed true positive. False positive dry-run: 14 days of production monitoring produced 23 total matches. Of these, 19 were from the three previously baselined service accounts (excluded in tuning), 2 were from a newly discovered monitoring tool (added to exclusion list), and 2 were from a legitimate administrative action (documented as benign true positives). Post-tuning false positive rate: approximately 8%. Below threshold. Approved for production.
Addressing the Objections
“We don’t have time for this level of process.”
You do not have time not to do this. Every hour spent on upfront engineering saves multiples in ongoing triage costs, tuning firefights, and incident response failures. The UCDL is not bureaucracy — it is the elimination of waste. Start with Phase 2 (feasibility) and Phase 4 (validation) alone, and the impact will be immediately visible.
“Our team is too small.”
The UCDL scales down. A team of two can implement a lightweight version: a shared backlog in a kanban board, peer review via pull requests, and a monthly review cadence instead of quarterly. The process adapts to team size; the principles do not change.
“We use vendor-provided rules. We don’t write custom detections.”
Vendor rules are a starting point, not a finished product. Every vendor rule — including Microsoft Sentinel’s built-in analytic templates — must be tuned for your specific environment. The UCDL applies equally to customizing vendor content as it does to building from scratch. In fact, the feasibility and validation phases are even more critical for vendor rules, because they were written for a generic environment that may not match yours.
“We’ll adopt this when we get a bigger budget.”
The UCDL requires no new tools. It requires discipline and a text editor. Version control is free (GitHub, GitLab). Attack simulation tools like Atomic Red Team are open source. The investment is in process and culture — neither of which requires a purchase order.
Conclusion: From Detection Rules to Detection Products
The cybersecurity industry has spent the last decade investing in better detection platforms — more powerful SIEMs, faster XDR engines, broader data ingestion pipelines. These investments are necessary, but they are not sufficient. A Formula 1 engine installed in a car with no steering system will crash on the first corner.
Detection content is the steering system. And right now, most organizations are building that steering system with the rigor of a weekend hobby project.
Tuned by Design is a call to change that. It is a call to treat detection use cases as engineered products — with requirements, designs, tests, measurements, and retirement plans. It is a call to adopt the same lifecycle discipline that transformed software development and apply it to the domain where it matters most: the detection layer that stands between attackers and the assets we are sworn to protect.
The Use Case Development Lifecycle is not a luxury for elite SOCs with unlimited budgets. It is a survival strategy for any security team that refuses to accept alert fatigue, blind spots, and “we had a rule but it was disabled” as inevitable outcomes.
“We stopped building software without a development lifecycle. It is time we stop building detections without one. Tuned by Design is not a framework you adopt. It is a standard you hold yourself to.”
The tools are ready. The platforms are capable. The question is whether we, as detection engineers, are willing to hold ourselves to the same engineering standard that the rest of the technology industry adopted years ago.
Tuned by Design: Why Detection Engineering Needs Its Own Development Lifecycle was originally published in Detect FYI on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.
1 post - 1 participant
Malware Analysis, News and Indicators - Latest topics
Sp123
"An underestimated security threat to organizations is employee apathy and burn out."
