If your company builds, integrates, or operates AI for a federal customer, someone is eventually going to ask how you tested it. Not whether the model is accurate. Whether it can be made to do something it should not.
Most defense contractors have people who can answer that question about a web application and nobody who can answer it about an LLM. The gap is narrower than it looks.
What Is Actually Driving the Requirement
Be precise about this, because vendors are not. No regulation currently says federal contractors must red team their AI systems. What exists is a stack of guidance and contract language moving in one direction.
NIST’s AI Risk Management Framework (AI 100-1) treats adversarial testing as part of the Measure function. NIST SP 800-218A extends the Secure Software Development Framework to generative AI, which matters because SSDF attestation already appears in federal software procurement. For companies handling CUI, the DFARS 252.204-7012 and CMMC obligations do not mention AI specifically, but an AI system that leaks CUI through model output is still a CUI spill.
The practical trigger is usually simpler than any of that. A program office asks for evidence of adversarial testing, and the contractor discovers nobody on staff has done it.
The Curriculum That Matters
Skip anything organized around “AI awareness.” The training that produces capability is organized around attack classes and mapped to a taxonomy the government already recognizes: MITRE ATLAS.
- Prompt injection, direct and indirect (ATLAS AML.T0051, OWASP LLM01). The direct case is a chat box. The interesting case for contractors is indirect: an instruction hidden in a document that a retrieval pipeline pulls in. If your system ingests contractor-submitted files, vendor documentation, or open-source intelligence, that is your attack surface.
- Jailbreaking (ATLAS AML.T0054). Pushing a model past its safety training and documenting which technique worked, so the finding survives the next model update.
- Model evasion (ATLAS AML.T0015). Crafting input that a deployed classifier misses. For anyone shipping ML-based detection to a government customer, this is the finding that changes deployment decisions.
- Training data poisoning. Where the training or fine-tuning data comes from, who can influence it, and what a supply chain review of a model actually looks at.
- Exfiltration through inference. Whether the system surfaces its system prompt, its training data, or the contents of connected data sources under sustained multi-turn pressure.
The architecture half is what separates useful training from a payload list. Attacks land in the plumbing: how retrieved context gets concatenated into a prompt, what a tool-calling agent is permitted to invoke, whether the vector store enforces the same access controls as the system of record. A course that never leaves the chat interface teaches half the problem.
Tooling, and Why Scanners Are Not Enough
A serious syllabus names its tools. garak from NVIDIA runs probe suites across known injection and jailbreak payloads and gives you a baseline. PyRIT orchestrates multi-turn attacks where the payload assembles across a conversation. promptfoo turns confirmed findings into a regression suite so a prompt change does not silently reopen a hole.
In our labs students run the scanner first, then script their own attacks for what it missed. The habit matters more than the tool. An assessment that is only a garak report is the AI equivalent of submitting raw Nessus output as a penetration test, and a government reviewer will read it that way.
Delivery Constraints Are the Hard Part
For most contractors the curriculum is the easy question. Logistics is where training procurement dies.
Cleared personnel often cannot take a hosted course, cannot install a lab client on a government-furnished laptop, and cannot reach a commercial model API from the network where the work happens. The answer is a self-contained lab: everything inside a guest image, local models instead of hosted ones.
When we ran a three-day AI course for a military cyber unit, the entire engagement executed with no external network. Labs ran in the Centaur VM, adversarial exercises hit local models served by Ollama, and no request left the unit’s environment. Twenty-five operators spent day two attacking models and day three building agents. The logistics detail that bites people: model weights have to be staged before the environment goes behind the air gap, because ollama pull needs a network.
Instructor eligibility is the other constraint worth asking about early. GTK Cyber instructors carry federal program experience including DARPA and ARPA-H AI support, and one holds an active TS clearance with full-scope polygraph. Ask any vendor this question before you scope a delivery, not after.
Where This Training Does Not Help
Two honest limits.
It will not make your company an AI assurance shop. A two-day course produces staff who can run a competent assessment against an LLM application and write it up defensibly. It does not produce people who can evaluate a frontier model’s alignment properties or do original adversarial ML research. If your contract scope is model evaluation at that depth, you are hiring, not training.
It also will not satisfy a compliance requirement by itself. Training builds the capability; the deliverable a program office wants is a tested system and a report. Contractors who buy training expecting an artifact are disappointed. Buy it because you need people who can produce the artifact.
GTK Cyber’s AI Red-Teaming course runs as a closed-cohort engagement for federal and defense teams, on-site or virtual, with the offline lab environment described above. Registration data, NAICS codes, and past performance are on the government page if you are building a sources sought response or a subcontractor package.