SigmaHQ Quality Assurance Pipeline
How we ensure community-contributed rules are up to par
Photo by Sigmund on Unsplash
Here is a fact for you, the SigmaHQ rule repository is one of the biggest vendor agnostic repository for detection rules today, with hundreds of yearly contributions and thousands of detection rules to maintain, hence, setting a standard is a requirement.
In this blog I will be showcasing how the SigmaHQ rule repository set this standard via various tooling and CI pipelines, in order to make sure that rule quality is consistent and deployment ready. Let’s get started.
It all starts with the specification
The first step in setting a standard for rule writing is actually defining that standard. We host a repository called Sigma-Specification where we describe exactly the constituents of a Sigma rule and its different type.
GitHub - SigmaHQ/sigma-specification: Sigma rule specification
The specification goes into details on what fields are required vs optional, how do these fields have to be written, for example either lists or strings. We also define what are the allowed tags, modifiers and taxonomy of field names.
All of this helps define how a rule should look like at a minimum, and avoid any confusion while writing them. While the Sigma specification aims to offer a maximum of compatibility, flexibility and customization (you can add all the meta data you want), offering a big rule repository as well, we need to be more strict there.
An Additional Layer of Enforcement — SigmaHQ Convention
The SigmaHQ rule repository is technically an implementation of the specification and targeted towards a much larger community of contributors. Therefore we extend the specification with a bunch of conventions that make sure all rules look the same at a format level. These conventions are only applicable to the rules repository and shouldn’t be confused with the Sigma specification which is only a subset of the requirements imposed on the rules repository.
These conventions can be found in the same specification repo in a dedicated folder.
sigma-specification/sigmahq at main · SigmaHQ/sigma-specification
Currently we define 3 types of conventions.
- Filename Convention - That ensures that files can easily be identified via their filename and sorted accurately in their respected folders according to the logsource they target.
- Title Convention - This convention ensures that rules do not have random titles but instead, the aim is convey exactly what the rule is detecting by using the appropriate language
- Rule Convention - Basically enforces certain aspects that the general specification leaves either optional or not defined intentionally. Such as the expected indentation the actual order of the fields, and how should a description, false positives sections should look like.
Validators Are Here To Validate
Having all of that defined in a repository somewhere is nice and all, but how do we make sure its enforced? The answer is CI.
Sigma-Rules-Validator Action
The first piece of CI we use is called sigma-rules-validator which takes the specification JSON schemas we provide and validate all rules against them via check-jsonschema
This GitHub Action is available here for anyone to use
GitHub - SigmaHQ/sigma-rules-validator: Validates Sigma rules using the JSON schema
PySigma Core Validators
As we’ve described above, the specification defines certain rules about required fields, the way certain field should be written, etc. These rules are coded into the sigma library pySigma into something called Validators .
pySigma/sigma/validators/core at main · SigmaHQ/pySigma
If you leverage the Sigma Command Line Interface sigma-cli you can call validate those rules via the sigma check command.
Some of these validators include
- Identifier Existence.
- Duplicate Title.
- Duplicate Reference.
- Invalid Tag Format.
- Dangling Conditions.
PySigma SigmaHQ Validators
For those astute readers, you might ask yourself what about the SigmaHQ convention how are those enforced, since they are not part of the specification.
Well, for that we created a new repository called pySigma-validators-sigmaHQ that extends the list of validators with ones that enforce the conventions we define at SigmaHQ rule repo.
GitHub - SigmaHQ/pySigma-validators-sigmaHQ
These might include:
- Filename enforcement.
- Field <-> Logsource Check.
- Presence of certain tags.
Good Log Validation
Now all of the syntax and conventions are validated, time to validate the logic.
One of the tests we integrated a couple of years ago is what we call a good-log test. The idea is simple at its core. Test the sigma logic against benign logs where if a match occurs than we know the logic needs updating.
For this we use a dataset that contains benign system process and every day application execution. The dataset is stored in the following repository.
We currently test against the following baselines.
- Windows 10 Client.
- Windows 11 Client.
- Windows 7.
- Windows Server 2022 Base.
- Windows Server 2022 With AD Role.
- Windows Server 2022 Azure Image.
- Windows 11 Client.
You can read more on how the data was generated in the repo itself. We highly encourage contributions ;)
We leverage the evtx-sigma-checker binary to test sigma rules against these logs. The example below showcase the CI job for testing against the Windows 7 dataset.
check-baseline-win7:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Download evtx-sigma-checker
run: wget --no-verbose https://github.com/NextronSystems/evtx-baseline/releases/download/$EVTX_BASELINE_VERSION/evtx-sigma-checker
- name: Download and extract Windows 7 32-bit baseline
run: |
wget --no-verbose https://github.com/NextronSystems/evtx-baseline/releases/download/$EVTX_BASELINE_VERSION/win7-x86.tgz
tar xzf win7-x86.tgz
- name: Check for Sigma matches in baseline
run: |
chmod +x evtx-sigma-checker
./evtx-sigma-checker --log-source tests/thor.yml --evtx-path win7_x86/ --rule-path rules/windows/ --rule-path rules-emerging-threats/ --rule-path rules-threat-hunting/ > findings.json
- name: Show findings excluding known FPs
run: |
chmod +x .github/workflows/matchgrep.sh
./.github/workflows/matchgrep.sh findings.json .github/workflows/known-FPs.csv
Regression Testing
The most recent addition to our pipelines is regression testing. Here we validate a rule against a “malicious” log where we are expecting a match. We use this in order to make sure that a rule is always working as it supposed to.
Currently we support only matching against EVTX but this will be extended to other types in the future.
The idea is very simple. A sigma rule contain a regression_tests_path field that points to an info.yml file.
id: 6f781d8b-1b6c-408b-a90d-08aceb2a14d0
description: N/A
date: 2025-10-24
author: SigmaHQ Team
rule_metadata:
- id: d7662ff6-9e97-4596-a61d-9839e32dee8d
title: Add SafeBoot Keys Via Reg Utility
regression_tests_info:
- name: Positive Detection Test
type: evtx
provider: Microsoft-Windows-Sysmon
match_count: 1
path: regression_data/windows/process_creation/proc_creation_win_reg_add_safeboot/d7662ff6-9e97-4596-a61d-9839e32dee8d.evtx
This file will contain a list pointing to the logs we have to match against. In the example above we only have one.
The CI will then parse this and call evtx-sigma-checker (because of type: evtx ).
jobs:
true-positive-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install Python dependencies
run: |
python -m pip install --upgrade pip
pip install pyyaml
- name: Download evtx-sigma-checker
run: |
wget --no-verbose https://github.com/NextronSystems/evtx-baseline/releases/download/$EVTX_BASELINE_VERSION/evtx-sigma-checker
chmod +x evtx-sigma-checker
- name: Run regression tests
run: |
python tests/regression_tests_runner.py --rules-paths "rules/,rules-emerging-threats/,rules-threat-hunting/" --evtx-checker ./evtx-sigma-checker --thor-config tests/thor.yml --ignore-validation
Four-Eyes Principle
While we’ve concluded the automated part of the Sigma validation, one final part that is equally as important is the reviewers actually looking at the rules. This has always been an important part of SigmaHQ.
We apply a four eye principal meaning we require at least 2 reviewers in order to merge a PR. (except in some cases, we like bureaucracy only so much).
These reviewers are some of the best in terms of detection engineering knowledge and with varying level experiences and expertise.
And its not about just looking at raw YAMLs
- We strive to replicate most attacks that are contributed (especially on Windows) to make sure logic is reflected in the logs.
- We investigate linked references to their fullest and often pivot via utilities like VT or ANY.RUN to find more samples leveraging the same techniques and help extend coverage in the same PR.
Let me just say that its more than just a “LGTM” review ;)
Conclusion & Future Plans
Hope this gave you a better understanding on how Sigma strive to maintain high quality of rules.
We’re always trying to improve. Happy to hear any feedback or suggestions. Hit me up on socials Twitter / LinkedIn or join the discussion on our Sigma discord group.
Join the SigmaHQ Community Discord Server!
A lot of exciting automation and additional features are coming in the next few months that will elevate the quality and trust in rules and open source to the next level. I will keep you posted :D
SigmaHQ Quality Assurance Pipeline was originally published in Sigma_HQ on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.
Article Link: SigmaHQ Quality Assurance Pipeline | by Nasreddine Bencherchali | Nov, 2025 | Sigma_HQ
1 post - 1 participant
Malware Analysis, News and Indicators - Latest topics
Sp123
"The real threat is actually not when the computer begins to think like a human, but when humans begin to think like computers."