Key Concepts¶
Scenario¶
A text description of infrastructure to create. Examples:
- "Create an S3 bucket for healthcare PHI data with HIPAA compliance"
- "Create a VPC with public and private subnets and NAT gateway"
- "Create IAM roles for a CI/CD pipeline with least privilege"
114 scenarios across 18 domains. Use adversarial-iac scenarios to browse.
Difficulty¶
Controls vulnerability complexity and stealth level. Difficulty adjusts how hard it is for Blue Team to detect vulnerabilities — not how many mistakes Red Team makes.
| Level | Vulns | Style |
|---|---|---|
| Easy | 1-2 | Obvious misconfigs (missing encryption, public access). Minimal stealth. |
| Medium | 2-3 | Moderate stealth. Vulnerabilities require basic cross-resource analysis. |
| Hard | 1-2 | High-stealth, logic-dependent. Requires multi-resource reasoning, conditional logic, cross-file dependencies. Focus on complexity, not quantity. |
Difficulty ≠ error rate
Red Team is never asked to "make mistakes." Difficulty controls vulnerability sophistication and stealth, not sloppiness. Manifests must be accurate at all levels.
Manifest¶
The Red Team's secret list of what they hid. Stored in red_team_manifest.json. Each entry has vulnerability type, location, and evidence. The Judge uses this as ground truth.
Findings¶
The Blue Team's reported issues. Stored in blue_team_findings.json. Each has severity, location, evidence, and remediation. The Judge matches these to manifest entries.
Scoring Terms¶
| Term | Definition |
|---|---|
| TP (True Positive) | Blue correctly found a Red vulnerability |
| FP (False Positive) | Blue reported something not in the manifest |
| FN (False Negative) | Red vulnerability that Blue missed |
| Precision | TP / (TP + FP) — Blue's accuracy |
| Recall | TP / (TP + FN) — % of vulns Blue found |
| F1 | 2 × (P × R) / (P + R) — balanced score |
| Evasion Rate | FN / Total — Red's success at hiding |
| Phantom Concordance | Two LLMs agreeing on a non-existent vulnerability due to shared training distribution |
| Manifest Accuracy | % of Red Team claims confirmed by static tools (Trivy/Checkov) |
Precision Verification (v2.3)¶
Non-reasoning models are structurally constrained to 60–77% precision. The precision verification filter (--precision-strategy precise) adds a mandatory second pass: each finding must cite the exact code line causing the vulnerability. Findings without concrete evidence are removed before scoring.
Phantom Concordance (v2.3)¶
When both Red Team (generating ground truth) and Blue Team (detecting) are LLMs, shared training data can cause them to agree on vulnerabilities that don't actually exist. This inflates recall. The framework mitigates this through:
- Manifest validation gate — Trivy/Checkov must corroborate Red Team claims
- Cross-provider consensus — Novel vulnerability matches require agreement from models trained by different organizations