Ask a vendor whether their product uses machine learning and you will get a yes. Ask for the model architecture and you will get a slide. Neither answer tells you what is running in the detection path.
You do not need the vendor’s cooperation to find out. Four tests, run from outside the box on data you control, will tell you whether the thing scoring your events has learned parameters or is a weighted condition list with an AI label on the box. None of them require the vendor to open the model.
A Rule Engine Is Not the Problem
If a product scores an event by summing weights across 30 conditions, that can be a good detection engine. Rules are readable, predictable, and tunable by an analyst at 3 a.m. Plenty of detection problems should be solved that way.
The problem is paying for a model and receiving rules. You get neither the generalization to unseen variants that a trained model buys you nor the transparency that makes a rule pack cheap to operate. So the question worth answering is narrow: does the scoring function have parameters that were fit to data, or thresholds that a person typed in?
Test 1: Count the Unique Scores
You should already be insisting on raw per-detection output rather than dashboard counts as part of POC discipline. That raw stream answers this question in one line of pandas.
import pandas as pd
scores = pd.read_json('vendor_detections.jsonl', lines=True)['risk_score']
print(scores.nunique(), 'unique values across', len(scores), 'events')
print(scores.value_counts().head(10))A model with fit parameters produces a near-continuous distribution: hundreds or thousands of distinct values, most of them ugly decimals. A weighted rule engine produces a short repeating list, because every score is a sum over the same fixed condition set. Twelve unique values across 8,000 events is a rubric, not a model. Scores piling up on 25, 50, 75, and 90 point the same direction.
The honest caveat: a vendor can bucket a real model’s output before it reaches the API, which makes a genuine model look like a rubric. So treat low cardinality as a prompt to ask whether the pre-bucketing score is exposed, then run the next test, which bucketing cannot hide.
Test 2: Sweep One Feature and Watch the Boundary
This is the test that settles it. Take an event the product flags, vary a single field in small increments, resubmit each variant, and record the score.
import numpy as np
rows = []
for length in range(8, 40):
for entropy in np.arange(2.0, 4.6, 0.1):
domain = synth_domain(length, entropy) # your generator
rows.append({'length': length,
'entropy': round(entropy, 1),
'score': vendor_api.score(domain)})
grid = pd.DataFrame(rows).pivot(index='length', columns='entropy', values='score')
print(grid)Read the surface. If the score is flat and then jumps in a vertical line at a round entropy value, identical for every domain length, a threshold fired. If the score rises gradually and the entropy at which it rises depends on the length, the scoring function learned an interaction between two features, which is something no analyst hand-codes.
We teach this same procedure as the model-extraction lab on day four of the Applied Data Science and AI for Cybersecurity course: query a black-box classifier through its API until the decision boundary becomes visible. The technique comes from the 2016 USENIX Security paper Stealing Machine Learning Models via Prediction APIs, and MITRE ATLAS catalogs the offensive version as Extract AI Model (AML.T0024.002). That matters procedurally: get written authorization and a rate limit agreed in advance, because a vendor’s abuse detection will read a high-volume sweep exactly the way ATLAS describes it.
Test 3: Ask for the Version String, Not the Architecture
Architecture answers cost a vendor nothing. Versioning is expensive to fake, because it only exists if someone built a pipeline. Ask for three artifacts:
- A model version in every detection payload.
model_version: "url-clf-2026.06.3"on the detection itself, not the product release number in the footer. - A retraining changelog. Dates, and the held-out evaluation metrics for each version. Redaction is fine. Absence is the answer.
- The drift monitor. What input distribution it watches, what threshold trips it, and what the team did the last time it tripped.
NIST’s AI Risk Management Framework (AI 100-1) puts exactly this under its MEASURE and MANAGE functions, so a vendor selling into regulated buyers has no excuse for being surprised by the request. The artifact to ask for by name is a model card, from Model Cards for Model Reporting (Mitchell et al., 2019): intended use, training data, evaluation results, known failure modes. Teams running real models usually have something like it internally. A product that has “used ML since 2019” and cannot produce a single retraining date is not maintaining a model.
Test 4: Read What Actually Ships
If any component runs in your environment, the inference stack is sitting on your disk. This is the fastest of the four and the hardest to spin.
- List the bundled runtime’s dependencies for an inference library:
onnxruntime,xgboost,lightgbm,torch,scikit-learn. - Look for serialized weights:
find /opt/vendor -name '*.onnx' -o -name '*.pt' -o -name '*.pkl' -o -name '*.joblib'. - Run
stringsandlddagainst native binaries for ONNX Runtime or libtorch symbols. - Check egress. If the product claims local inference but the agent opens a connection to a scoring endpoint before every verdict, the model is not local, whatever the datasheet says.
The ratio is the finding. A 400 KB gradient-boosting model next to a 40,000-line YAML rule pack tells you which component does the work.
What These Tests Do Not Tell You
None of this measures whether the product is any good. It measures whether one specific claim is true. A tuned rule pack from a vendor with deep threat-intel coverage will outperform a poorly trained model on your traffic, and for some vendors the rule pack is the genuinely valuable asset. Establish what the engine is so you can price it, then judge it on detection lift and false positive cost against your current stack.
These four tests also do not transfer to LLM-wrapper products. When the “AI” is a hosted model behind a prompt, there is no decision boundary to sweep and score cardinality means nothing. The questions there are which model, what grounding, what happens to the output at temperature above zero, and whether the prompt is a trust boundary.
Reading a score distribution and probing a decision boundary are ordinary data science skills, which is the point: the technical literacy to test a vendor claim is the same literacy that lets your team build detections. The executive AI course covers the decision side of this for security leaders, and the applied course is where analysts write the probe scripts themselves. Both sit inside GTK Cyber’s AI cybersecurity training track. For the questions to open the conversation with, start with our vendor evaluation checklist.