This resource is brought to you courtesy of Tejas Cyber Network (Founder - Anshu Gupta) which is a global, OPEN TO ALL, vendor agnostic, professional networking community of cybersecurity leaders, under its Project S (S stands for Skills) initiative.
Join Us - Cyber Practitioners, Cyber Founders
Follow us - LinkedIn, Subscribe to Tejas Events Calendar
Upcoming - Sat, Oct 24th - Global CISO Summit - Bay Area Edition (Sponsorship Interest Form)
A curated list of AI security and safety evaluation benchmarks well-regarded by Frontier AI labs (Anthropic, OpenAI, Google DeepMind, Meta) and AI Safety Institutes (US AISI, UK AISI).
Maintained by: Anshu Gupta, Founder, Tejas Cyber Network
GitHub: https://github.com/anshug/ai-security-evals
| Category | Count |
|---|---|
| Cyber Offense and CTF | 9 |
| Cyber Defense and Threat Intel | 8 |
| Software Security and Code | 2 |
| Agent Security and Prompt Injection | 8 |
| Jailbreak and Refusal | 6 |
| CBRN Knowledge and Bio Uplift | 9 |
| Alignment, Honesty, Scheming | 4 |
| Autonomy and AI R&D | 4 |
| Comprehensive Safety and Trust | 6 |
| Bias and Fairness | 1 |
| Tooling Frameworks | 4 |
| Benchmark | Category | Organization | Cited By | Original List | Paper |
|---|---|---|---|---|---|
| Cybench | Cyber Offense and CTF | Stanford / academic consortium | Anthropic, OpenAI, Google DeepMind, UK AISI | Yes | Link |
| Cybergym | Cyber Offense and CTF | UC Berkeley | Anthropic (Opus 4.7 system card), OpenAI (GPT-5.5 system card) | Yes | Link |
| CVEBench | Cyber Offense and CTF | Academic | Used in academic and industry evaluations | Yes | Link |
| 3CB (Catastrophic Cyber Capabilities Benchmark) | Cyber Offense and CTF | Apollo Research / Apart Research | Apollo Research, UK AISI, academic literature | No | Link |
| SOLVE (Scoring Obstacle Levels in Vulnerabilities & Exploits) | Cyber Offense and CTF | Irregular (Pattern Labs) | Used by Irregular in frontier lab assessments | Yes | Link |
| CyScenarioBench | Cyber Offense and CTF | Irregular (Pattern Labs) | Used by Irregular in frontier lab assessments | Yes | Link |
| NYU CTF Bench | Cyber Offense and CTF | NYU | Academic, frontier lab cyber suites | No | Link |
| HonestCyberEval / AIxCC | Cyber Offense and CTF | Alan Turing Institute / DARPA | Alan Turing Institute, DARPA | No | Link |
| XBOW internal benchmark | Cyber Offense and CTF | XBOW | XBOW, cited in OpenAI Daybreak / GPT-5.5 system card discussions | No | Link |
| CTI-REALM | Cyber Defense and Threat Intel | Microsoft | Microsoft Security Research | Yes | Link |
| ExCyTIn-Bench | Cyber Defense and Threat Intel | Microsoft | Microsoft Security Research | Yes | Link |
| CyberSOCEval | Cyber Defense and Threat Intel | Meta (with CrowdStrike) | Meta, CrowdStrike | Yes | Link |
| CTIBench | Cyber Defense and Threat Intel | Academic (Alam et al.) | Cisco Foundation-Sec, academic security LLM evals | Yes | Link |
| CyberMetric | Cyber Defense and Threat Intel | Technology Innovation Institute (TII) / Khalifa University | Foundation-Sec models, academic security LLM evaluations | No | Link |
| SecBench | Cyber Defense and Threat Intel | Tencent / HK PolyU | Tencent, academic security LLM evals | No | Link |
| SecEval | Cyber Defense and Threat Intel | Academic (Li et al.) | Cisco Foundation-Sec, academic security LLM evals | No | Link |
| SecQA | Cyber Defense and Threat Intel | Academic | Academic security LLM evals | No | Link |
| SusVibes | Software Security and Code | Academic | Academic / under publication | Yes | Link |
| CyberSecEval (Meta Purple Llama, v1 to v4) | Software Security and Code | Meta | Meta (Llama 4 system card), applied to OpenAI, Google, Anthropic models | No | Link |
| Agent Red Teaming (ART) | Agent Security and Prompt Injection | Gray Swan AI | Gray Swan, frontier lab agentic evals | Yes | Link |
| SHADE-Arena | Agent Security and Prompt Injection | Anthropic | Anthropic | Yes | Link |
| AgentDojo | Agent Security and Prompt Injection | ETH Zurich / Invariant Labs | US AISI, UK AISI, NeurIPS 2024 SafeBench prize winner | No | Link |
| InjecAgent | Agent Security and Prompt Injection | UIUC (Kang Lab) | Widely cited, used in frontier agent security research | No | Link |
| AgentHarm | Agent Security and Prompt Injection | Gray Swan AI / UK AISI | Gray Swan, UK AISI, ICLR 2025 | No | Link |
| BIPIA | Agent Security and Prompt Injection | Microsoft | Microsoft Security Research | No | Link |
| Tensor Trust | Agent Security and Prompt Injection | UC Berkeley | Referenced in AgentDojo, OpenAI prompt injection work | No | Link |
| BrowserART | Agent Security and Prompt Injection | Gray Swan AI | Gray Swan, frontier browser agent evals | No | Link |
| StrongREJECT | Jailbreak and Refusal | UC Berkeley | OpenAI, Anthropic system cards | Yes | Link |
| HarmBench | Jailbreak and Refusal | Center for AI Safety (Mazeika et al.) | Anthropic, OpenAI, Google DeepMind, Meta system cards | No | Link |
| JailbreakBench | Jailbreak and Refusal | Academic (Chao, Debenedetti, Robey, et al.) | Anthropic, OpenAI, academic safety research | No | Link |
| XSTest | Jailbreak and Refusal | Academic (Rottger et al.) | OpenAI, Anthropic, Google DeepMind system cards | No | Link |
| SORRY-Bench | Jailbreak and Refusal | Princeton / Virginia Tech | Academic safety research | No | Link |
| AdvBench | Jailbreak and Refusal | CMU / Center for AI Safety | Widely cited across frontier labs | No | Link |
| WMDP (Weapons of Mass Destruction Proxy) | CBRN Knowledge and Bio Uplift | Center for AI Safety + Scale AI consortium | Anthropic, OpenAI, Google DeepMind, Amazon Nova, Meta | No | Link |
| LAB-Bench | CBRN Knowledge and Bio Uplift | FutureHouse | Anthropic, OpenAI, Amazon Nova system cards | No | Link |
| Virology Capabilities Test (VCT) | CBRN Knowledge and Bio Uplift | SecureBio | Anthropic, OpenAI, frontier CBRN sections | No | Link |
| Long-form Biorisk Questions (LFB) | CBRN Knowledge and Bio Uplift | Gryphon Scientific (now Deloitte) | OpenAI Preparedness Framework evaluations | No | |
| Tacit Knowledge and Troubleshooting (TTK) | CBRN Knowledge and Bio Uplift | Gryphon Scientific (now Deloitte) | OpenAI Preparedness Framework evaluations | No | |
| Creative Biology (CrB) | CBRN Knowledge and Bio Uplift | SecureBio | Anthropic system cards | No | |
| Short-Horizon Bio Tasks (SHB) | CBRN Knowledge and Bio Uplift | Faculty.ai / Anthropic | Anthropic system cards | No | |
| FORTRESS | CBRN Knowledge and Bio Uplift | Scale AI | Scale AI, frontier CBRN evaluations | No | Link |
| MOCET (Monte Carlo Expected Threat) | CBRN Knowledge and Bio Uplift | Johns Hopkins School of Medicine | Academic CBRN risk methodology | No | Link |
| MASK Benchmark | Alignment, Honesty, Scheming | Center for AI Safety | Anthropic, OpenAI safety research | No | Link |
| Apollo In-Context Scheming Evaluations | Alignment, Honesty, Scheming | Apollo Research | Apollo Research, used pre-deployment by Anthropic and OpenAI | No | Link |
| Stealth and Situational Awareness Evaluations | Alignment, Honesty, Scheming | Google DeepMind / Apollo Research | Google DeepMind Frontier Safety Framework | No | Link |
| Situational Awareness Dataset (SAD) | Alignment, Honesty, Scheming | Laine et al. (academic) | Academic safety research, Apollo Research | No | Link |
| METR RE-Bench | Autonomy and AI R&D | METR (Model Evaluation and Threat Research) | OpenAI (o3, o4-mini, GPT-4.5, GPT-5.1), Anthropic (Claude 3.7+), White House NSM on AI, EU AI Act | No | Link |
| METR HCAST / Task-Length Suite | Autonomy and AI R&D | METR | Used by METR for pre-deployment evals of OpenAI and Anthropic models | No | Link |
| MLAgentBench | Autonomy and AI R&D | Stanford | Anthropic, academic AI R&D evals | No | Link |
| SWE-bench Verified | Autonomy and AI R&D | OpenAI Preparedness / Princeton NLP | Anthropic, OpenAI, Google DeepMind system cards | No | Link |
| AILuminate v1.0 | Comprehensive Safety and Trust | MLCommons (AIRR Working Group) | MLCommons consortium, Stanford AI Index Report | No | Link |
| AIR-Bench 2024 | Comprehensive Safety and Trust | Stanford CRFM | Stanford CRFM, frontier safety research | No | Link |
| DecodingTrust | Comprehensive Safety and Trust | UIUC / Stanford / Berkeley | Widely cited across frontier labs and academic safety | No | Link |
| TrustLLM | Comprehensive Safety and Trust | Academic consortium | Academic, frontier safety research | No | Link |
| SafetyBench | Comprehensive Safety and Trust | Tsinghua University | Academic, multilingual safety evals | No | Link |
| WalledEval | Comprehensive Safety and Trust | Walled AI Labs | Industry safety platforms | No | Link |
| Bias Benchmark for Question Answering (BBQ) | Bias and Fairness | NYU (Parrish et al.) | Anthropic (Opus 4.5/4.6/4.7 system cards), OpenAI, academic fairness research | Yes | Link |
| UK AISI Inspect Evals | Tooling Frameworks | UK AI Safety Institute | US AISI, UK AISI, joint frontier model red-teaming | No | |
| MITRE ATLAS | Tooling Frameworks | MITRE | MITRE, NIST AI RMF, industry adoption | No | |
| Microsoft PyRIT | Tooling Frameworks | Microsoft | Microsoft AI Red Team | No | |
| Garak | Tooling Frameworks | NVIDIA | NVIDIA, industry red-teaming | No | Link |