Testing AI Systems for Security Vulnerabilities: Start Below the Model

In March 2024, Oligo Security disclosed that attackers had been submitting jobs to internet-exposed Ray clusters, using the access to lift cloud credentials and run cryptominers on GPU nodes. The entry point was Ray’s Jobs API, which has no authentication. The CVE, CVE-2023-48022, is disputed: Anyscale’s position is that Ray was never meant to be reachable from an untrusted network.

Both positions are correct, and nobody sent a single prompt.

That incident is a better syllabus for testing AI systems than most courses sold under that name. The training market has converged on jailbreaks and prompt injection because they demo well on a projector. The CVE record points somewhere less glamorous.

What 506 AI CVEs Actually Contain

This site publishes a monthly sync of AI and LLM vulnerabilities from NVD, filtered to CVSS 4.0 and above, at /cve/. The September 2026 snapshot holds 506 records. Counting weakness classes across them:

MLflow alone accounts for 77 records, 15 of them rated critical. CVE-2023-6018 lets an unauthenticated attacker overwrite any file on the tracking server (CVSS 9.8). CVE-2023-1177 is a path traversal fixed in 2.2.1. None of this requires knowing what a transformer is.

The 147 records tagged with prompt injection make the same point from the other side. Read the descriptions and the injection is almost always the delivery mechanism, with the score coming from the sink. Vanna’s ask method executed model-generated Python, so a crafted question became code execution (CVE-2024-5565, CWE-94). An IDE assistant could be steered by indirect injection into reading files outside the project (CVE-2025-62356, CWE-22). An agent’s media tool fetched attacker-chosen URLs from inside the network (CVE-2026-28451, CWE-918). A tester who stops at “the model followed my instruction” has found the door and not the room behind it.

Enumerate the Serving Stack First

An AI assessment should open like any other internal test: find the services, then find the ones that trust the network. The AI stack has its own default ports, and several of them ship without authentication.

nmap -sV -p 5000,8000,8081,8265,8888,11434 10.20.0.0/16 -oA ml-stack

# Confirm each hit with a read-only request before touching anything else
curl -s http://10.20.4.17:8265/api/jobs/                  # Ray Jobs API: job list, entrypoints
curl -s http://10.20.4.22:11434/api/tags                  # Ollama: installed models
curl -s "http://10.20.4.30:5000/api/2.0/mlflow/experiments/search?max_results=5"
curl -s http://10.20.4.41:8000/v1/models                  # vLLM OpenAI-compatible server
curl -s http://10.20.4.50:8081/models                     # TorchServe management API

A 200 with a JSON body and no credentials is a finding before you ever look at a version string. It maps to AML.T0049 (Exploit Public-Facing Application) in MITRE ATLAS and T1190 in ATT&CK when it faces the internet.

Then look at what the GPU host carries. Inference and training nodes routinely hold a Hugging Face token, an S3 or GCS credential with write access to the model bucket, and an instance role. That is AML.T0055 (Unsecured Credentials) waiting to happen, and it is why ShadowRay’s operators went for credentials before the miners.

Watch the internal wiring too. vLLM’s Mooncake integration shipped pickle serialization over ZeroMQ sockets bound to every interface (CVE-2025-32444, CVSS 10.0, per the vLLM security advisory). A KV-cache transfer channel between inference nodes is not something a prompt-based test will ever touch. ss -tlnp on the node will.

The “Safe” Format Still Has a Loader

Teams that migrated from pickle checkpoints to GGUF or safetensors often mark the artifact problem closed. The tensors are inert. The loader is not always.

CVE-2024-34359 in llama-cpp-python (CVSS 9.6) came from the chat template stored in a GGUF file’s metadata. The library rendered it with an unsandboxed jinja2.Environment, so a model file from a public hub carried a server-side template injection payload that ran on load. The project advisory and fix moved rendering into Jinja’s sandbox. Inspecting the template before loading takes a few lines with the gguf package:

from gguf import GGUFReader

reader = GGUFReader("candidate-model.gguf")
field = reader.fields.get("tokenizer.chat_template")
if field:
    template = bytes(field.parts[field.data[0]]).decode()
    for marker in ("__class__", "__globals__", "__subclasses__", "import", "os."):
        if marker in template:
            print("suspicious template construct:", marker)

A legitimate chat template loops over messages and emits role tokens. It has no reason to reach for __globals__. This check is heuristic, the same way pickle scanners are, and the durable control is running the loader in a sandbox or a version that already sandboxes rendering. The broader checkpoint-triage workflow is in AI model security training.

What Training for This Should Look Like

A course that teaches testing AI systems for security vulnerabilities needs three layers, and most cover one:

• Infrastructure: fingerprinting Ray, MLflow, Ollama, vLLM, Triton, and Jupyter; testing their management APIs; tracing which credentials each node holds. This is conventional network and web testing against unfamiliar services.
• Artifacts: model files and their loaders, including metadata-driven code paths like the GGUF template case.
• Application and model behavior: prompt injection, tool abuse, RAG poisoning, and evasion. The workflow for that layer is in how to red team an LLM-powered application.

The test for any provider is simple: ask whether a lab puts an unauthenticated MLflow or Ray instance in front of you. If every exercise starts at a chat box, the course teaches one layer of three.

For the record on our own courses: GTK’s AI Red-Teaming course spends most of its lab time on the model and application layers (injection, exfiltration through model outputs, robustness evaluation) in the AI Training Dojo. It assumes you already test networks and web applications, because the infrastructure layer above is that skill set pointed at unfamiliar services.

Where This Framing Breaks

If your organization consumes models only through a hosted API, the serving stack belongs to the provider. Your exposure is the application layer, plus whatever your API keys can do, and time spent on port 8265 is time wasted.

Version scanning also misses the most important case. CVE-2023-48022 is disputed because the behavior is by design. A Ray cluster on a current release is still an unauthenticated job runner if it is reachable. Patch status tells you nothing there. Network placement tells you everything, which is why the enumeration step comes before the CVE lookup and not after it.