The developer adding a chat feature to an internal tool is now making security decisions that used to belong to an architecture review. What the model can call, what happens to its output, which credential the tool runs under: those choices are made in a pull request, usually in an afternoon, usually by someone whose AI security training consists of a slide deck about prompt injection.
Most AI security training on the market is built for the SOC. It teaches detection, triage, and data analysis, which is the correct curriculum for an analyst and the wrong one for the person writing the application. Here is what the developer version has to contain.
Map the architecture to named weaknesses first
A developer cannot secure an abstraction. The first exercise should be drawing the actual data path of the feature being shipped: user input, retrieved documents, system prompt, model, output, and every consumer of that output.
Then name the weaknesses against the OWASP Top 10 for LLM Applications. Naming matters for a practical reason: a finding filed as LLM06 Excessive Agency gets triaged, and a ticket that says “the AI might do something bad” does not.
A chat feature that only renders text into a page has one exposure. The same model wired to a tool that runs a database query has a different one. Same vendor, same API, different threat model, and no generic training can make that call for a team.
Output handling breaks first
The most common production bug is not a clever jailbreak. It is a developer treating model output as trusted because it came from their own API call.
# The model returns text. This treats it as code.
sql = llm.generate(f"Write SQL to answer: {question}")
cursor.execute(sql) # LLM05, and now also SQL injectionModel output is untrusted input to whatever consumes it. Rendering it as HTML gives you cross-site scripting with an extra hop. Passing it to a shell gives you command injection. Writing it into a downstream prompt gives you a chain nobody is auditing.
The pattern that holds is constraining the model to a choice rather than to code generation:
# The model picks from a fixed set. The application writes the query.
intent = llm.classify(question, labels=["revenue", "headcount", "churn"])
cursor.execute(QUERIES[intent], params) # parameterized, allowlistedLabs should make students exploit their own feature first. Reading about LLM05 Improper Output Handling does not land the way watching your own chatbot render an injected <img onerror=...> tag does.
Agency is an authorization problem, not a prompt problem
LLM06 Excessive Agency is where developer training diverges hardest from analyst training, because the fix is entirely in code the developer owns.
The rule: the model’s permission is the permission of the credential its tool calls run under. A system prompt saying “only read, never delete” is a suggestion to a text predictor. A database role with no DELETE grant is a control. Scope the token, scope the tool signature, require human confirmation for anything irreversible, and log tool invocations with the user identity that triggered them.
This is also where prompt injection stops being theoretical. An agent that reads a support ticket and can also call an email tool will eventually read a ticket written by someone who knows that. The offensive workflow for finding these paths is written up in red teaming an LLM-powered application. The defensive version is shorter: assume the instruction arrives, and make sure the tool it reaches cannot do real damage.
Tests run in CI or they do not run
A pentest before launch tells you about the prompt that existed that week. Prompts change weekly, and a one-line system prompt edit can reopen a bug that was closed in March.
Promptfoo fits a developer workflow because the cases live in version control next to the code:
redteam:
plugins: [pii, excessive-agency, hijacking]
strategies: [prompt-injection, jailbreak]
numTests: 20Garak gives broader probe coverage against the deployed endpoint. Both point at the application, not the base model, because the vulnerability is nearly always in the wiring. Treat a failed injection case the way you treat a failed unit test: it blocks the merge.
The limits worth stating
This curriculum does not make anyone a machine learning engineer, and it should not try. Securing an LLM feature is application security with an unusual input source. Teams that train or fine-tune their own models need the artifact and data-poisoning material covered in AI model security training, which is a different course for a different job.
It also will not help a team that has no authorization model to begin with. If tool credentials are shared service accounts with broad grants, the AI feature is not the problem that needs solving first.
We teach this material in the AI Cyber Bootcamp, where the red and blue team blocks cover evasion, poisoning, prompt injection, and RAG poisoning against live targets, and the labs run in a browser environment so nobody loses a morning to dependency installs. The course assumes you can read and modify code, which for this audience is a safe assumption.