Courses on AI Security Testing: Grade the Syllabus Against ATLAS

MITRE ATLAS had 16 tactics and 101 top-level techniques when our site last synced it on October 1, 2026. Of those 101, MITRE rates 44 as “realized,” meaning someone has used the technique against a production AI system, not a lab target. That number is the best tool you have for comparing courses on AI security testing and adversarial AI, because course outlines are written to sound alike and ATLAS IDs are not.

The method: get the lab list, tag each lab with technique IDs, count what is covered. It takes about an hour per course and it surfaces the gaps that a syllabus page hides.

Ask for Labs, Not Topics

A topic is something an instructor talks about. A lab is something you do with a target, a tool, and a result you can check. “Prompt injection” as a topic can mean a slide with a screenshot. As a lab, it means you wrote a payload, planted it in a retrieval source, and measured how often the model followed it.

So the first email to any provider is short: “Can you send the list of hands-on labs, with the tools and targets used in each?” Good providers answer in a day. If the answer is the same bullet list from the course page, you have learned something.

Then tag each lab. Use the most specific ATLAS ID you can defend. A few anchors that come up in nearly every syllabus:

• Direct and indirect prompt injection: AML.T0051.000 and AML.T0051.001, both under Execution.
• Jailbreaks: AML.T0054, which ATLAS files under both Privilege Escalation and Defense Evasion.
• System prompt extraction and data leakage: AML.T0056 and AML.T0057, both Exfiltration.
• Adversarial examples against a classifier: AML.T0043 (Craft Adversarial Data) for the staging step and AML.T0015 (Evade AI Model) for using it.
• Membership inference and model extraction: AML.T0024.000 and AML.T0024.002, sub-techniques of Exfiltration via AI Inference API.

If a lab does not map to anything, it is either process (reporting, scoping) or it is filler. Process labs are fine. Count them separately.

Score It in Python

The ATLAS data is published as a single YAML file in mitre-atlas/atlas-data. Every technique carries its tactic IDs and a maturity field, so coverage is a set operation:

import requests, yaml

URL = "https://raw.githubusercontent.com/mitre-atlas/atlas-data/main/dist/ATLAS.yaml"
matrix = yaml.safe_load(requests.get(URL, timeout=30).text)["matrices"][0]

tactics = {t["id"]: t["name"] for t in matrix["tactics"]}
techniques = {t["id"]: t for t in matrix["techniques"] if t["id"].count(".") == 1}

# One entry per lab the provider confirms students run, tagged with top-level IDs
labs = {
    "Indirect injection via a poisoned RAG document": ["AML.T0051"],
    "Jailbreak a guarded chat endpoint": ["AML.T0054"],
    "System prompt and context extraction": ["AML.T0056", "AML.T0057"],
    "Evasion attack on a malware classifier": ["AML.T0043", "AML.T0015"],
}

hit = {tid for ids in labs.values() for tid in ids}
covered = {tac for tid in hit for tac in techniques[tid].get("tactics", [])}

print(f"Tactics touched: {len(covered)}/{len(tactics)}")
for tid, name in tactics.items():
    print(f"  [{'x' if tid in covered else ' '}] {tid} {name}")

realized = [t for t in techniques.values() if t.get("maturity") == "realized"]
missed = sorted(f"{t['id']} {t['name']}" for t in realized if t["id"] not in hit)
print(f"\nRealized techniques with no lab: {len(missed)}/{len(realized)}")
print("\n".join(missed))

Against the October 1 data, those four labs touch 7 of 16 tactics and 3 of the 44 realized techniques. Four labs is about what a one-day LLM workshop fits. The tactic count looks respectable. The realized count tells the truth: the course teaches the part of AI attack surface that gets conference talks, and skips most of what has been used in the wild.

Swap in a second provider’s labs dict and diff the two missed lists. That diff is your comparison.

Where the Gaps Usually Are

The holes are predictable, because they follow from what is cheap to build as a lab. Four realized techniques rarely appear in an LLM-centric outline:

• AML.T0010 AI Supply Chain Compromise and AML.T0011.000 Unsafe AI Artifacts. A pickled PyTorch checkpoint executes code on load. A lab here means scanning a model with modelscan or decompiling the pickle with Trail of Bits’ fickling, then explaining why weights_only=True in torch.load matters. We covered why this layer produces most AI CVEs in testing AI systems from the serving stack up.
• AML.T0020 Poison Training Data and AML.T0018 Manipulate AI Model. Both sit under Persistence. If your organization retrains a phishing or fraud model on analyst-labeled data, whoever can influence those labels can plant a backdoor. The Adversarial Robustness Toolbox ships poisoning attacks (art.attacks.poisoning) that run against a scikit-learn model in a notebook.

The reason these get skipped is practical. Poisoning and supply chain labs need a training pipeline and model files, not a chat endpoint, so they are harder to build and harder to run in a browser. NIST AI 100-2 E2025, NIST’s taxonomy of adversarial ML, gives training-time attacks as much space as inference-time ones. Most courses do not.

Run It on Ours Too

Our AI Red-Teaming course page lists five topics. Mapped honestly, the first three land on Execution, Privilege Escalation, Defense Evasion, Exfiltration, and Impact (bias and harmful output fall under AML.T0048 External Harms). The last two topics, building red-team frameworks and compliance, are process and map to no technique. The page does not name a supply chain lab.

That is the point of the exercise. A topic list, ours included, is a summary written for a web page. Ask for the labs. In ours, students start with an automated scanner as a baseline and then spend most of their lab time writing their own multi-turn attack scripts. That detail is not on the course page. Pulling details like it out is what this audit is for. The AI red team training page has the fuller lab breakdown.

What the Score Does Not Measure

ATLAS coverage measures breadth, not depth. A course that spends a full day on indirect injection through tool calls (AML.T0053, AML.T0086) and scores 4 of 16 tactics can be worth more than one that touches 12 tactics with a slide each. Weight the tactics to your systems: LLM apps with retrieval and tools care about Execution and Exfiltration; ML detection pipelines care about AI Attack Staging, Defense Evasion, and Persistence.

The score also says nothing about whether the instructor can explain why an attack works, or whether students leave with code they can rerun. For that, read how to vet who teaches AI red-teaming. And skip the exercise entirely if you need LLM application testing only and the course is honest that LLM testing is all it covers. A narrow course that says it is narrow is fine. The audit is for catching the ones that claim “adversarial AI” and teach one tactic.