# How to Tell if an AI Security Tool Actually Uses Real AI

By Charles Givre · 2026-07-27

> Vendors will not show you the model. Four black-box tests that read score distributions, sweep the decision boundary, and check what the agent ships.

Ask a vendor whether their product uses machine learning and you will get a yes. Ask for the model architecture and you will get a slide. Neither answer tells you what is running in the detection path.

You do not need the vendor's cooperation to find out. Four tests, run from outside the box on data you control, will tell you whether the thing scoring your events has learned parameters or is a weighted condition list with an AI label on the box. None of them require the vendor to open the model.

## A Rule Engine Is Not the Problem

If a product scores an event by summing weights across 30 conditions, that can be a good detection engine. Rules are readable, predictable, and tunable by an analyst at 3 a.m. Plenty of detection problems should be solved that way.

The problem is paying for a model and receiving rules. You get neither the generalization to unseen variants that a trained model buys you nor the transparency that makes a rule pack cheap to operate. So the question worth answering is narrow: does the scoring function have parameters that were fit to data, or thresholds that a person typed in?

## Test 1: Count the Unique Scores

You should already be insisting on raw per-detection output rather than dashboard counts as part of [POC discipline](/blog/how-to-run-poc-ai-security-vendor). That raw stream answers this question in one line of `pandas`.

```python
import pandas as pd

scores = pd.read_json('vendor_detections.jsonl', lines=True)['risk_score']
print(scores.nunique(), 'unique values across', len(scores), 'events')
print(scores.value_counts().head(10))
```

A model with fit parameters produces a near-continuous distribution: hundreds or thousands of distinct values, most of them ugly decimals. A weighted rule engine produces a short repeating list, because every score is a sum over the same fixed condition set. Twelve unique values across 8,000 events is a rubric, not a model. Scores piling up on 25, 50, 75, and 90 point the same direction.

The honest caveat: a vendor can bucket a real model's output before it reaches the API, which makes a genuine model look like a rubric. So treat low cardinality as a prompt to ask whether the pre-bucketing score is exposed, then run the next test, which bucketing cannot hide.

## Test 2: Sweep One Feature and Watch the Boundary

This is the test that settles it. Take an event the product flags, vary a single field in small increments, resubmit each variant, and record the score.

```python
import numpy as np

rows = []
for length in range(8, 40):
    for entropy in np.arange(2.0, 4.6, 0.1):
        domain = synth_domain(length, entropy)          # your generator
        rows.append({'length': length,
                     'entropy': round(entropy, 1),
                     'score': vendor_api.score(domain)})

grid = pd.DataFrame(rows).pivot(index='length', columns='entropy', values='score')
print(grid)
```

Read the surface. If the score is flat and then jumps in a vertical line at a round entropy value, identical for every domain length, a threshold fired. If the score rises gradually and the entropy at which it rises depends on the length, the scoring function learned an interaction between two features, which is something no analyst hand-codes.

We teach this same procedure as the model-extraction lab on day four of the [Applied Data Science and AI for Cybersecurity](/courses/applied-data-science-ai) course: query a black-box classifier through its API until the decision boundary becomes visible. The technique comes from the 2016 USENIX Security paper [Stealing Machine Learning Models via Prediction APIs](https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer), and MITRE ATLAS catalogs the offensive version as [Extract AI Model (AML.T0024.002)](/atlas/AML.T0024.002). That matters procedurally: get written authorization and a rate limit agreed in advance, because a vendor's abuse detection will read a high-volume sweep exactly the way ATLAS describes it.

## Test 3: Ask for the Version String, Not the Architecture

Architecture answers cost a vendor nothing. Versioning is expensive to fake, because it only exists if someone built a pipeline. Ask for three artifacts:

- **A model version in every detection payload.** `model_version: "url-clf-2026.06.3"` on the detection itself, not the product release number in the footer.
- **A retraining changelog.** Dates, and the held-out evaluation metrics for each version. Redaction is fine. Absence is the answer.
- **The drift monitor.** What input distribution it watches, what threshold trips it, and what the team did the last time it tripped.

[NIST's AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) (AI 100-1) puts exactly this under its MEASURE and MANAGE functions, so a vendor selling into regulated buyers has no excuse for being surprised by the request. The artifact to ask for by name is a model card, from [Model Cards for Model Reporting](https://arxiv.org/abs/1810.03993) (Mitchell et al., 2019): intended use, training data, evaluation results, known failure modes. Teams running real models usually have something like it internally. A product that has "used ML since 2019" and cannot produce a single retraining date is not maintaining a model.

## Test 4: Read What Actually Ships

If any component runs in your environment, the inference stack is sitting on your disk. This is the fastest of the four and the hardest to spin.

- List the bundled runtime's dependencies for an inference library: `onnxruntime`, `xgboost`, `lightgbm`, `torch`, `scikit-learn`.
- Look for serialized weights: `find /opt/vendor -name '*.onnx' -o -name '*.pt' -o -name '*.pkl' -o -name '*.joblib'`.
- Run `strings` and `ldd` against native binaries for [ONNX Runtime](https://onnxruntime.ai/) or libtorch symbols.
- Check egress. If the product claims local inference but the agent opens a connection to a scoring endpoint before every verdict, the model is not local, whatever the datasheet says.

The ratio is the finding. A 400 KB gradient-boosting model next to a 40,000-line YAML rule pack tells you which component does the work.

## What These Tests Do Not Tell You

None of this measures whether the product is any good. It measures whether one specific claim is true. A tuned rule pack from a vendor with deep threat-intel coverage will outperform a poorly trained model on your traffic, and for some vendors the rule pack is the genuinely valuable asset. Establish what the engine is so you can price it, then judge it on detection lift and false positive cost against your current stack.

These four tests also do not transfer to LLM-wrapper products. When the "AI" is a hosted model behind a prompt, there is no decision boundary to sweep and score cardinality means nothing. The questions there are which model, what grounding, what happens to the output at temperature above zero, and whether the prompt is a trust boundary.

Reading a score distribution and probing a decision boundary are ordinary data science skills, which is the point: the technical literacy to test a vendor claim is the same literacy that lets your team build detections. The [executive AI course](/courses/executive-ai-guide) covers the decision side of this for security leaders, and the applied course is where analysts write the probe scripts themselves. Both sit inside GTK Cyber's [AI cybersecurity training](/lp/ai-cybersecurity-training) track. For the questions to open the conversation with, start with our [vendor evaluation checklist](/blog/evaluating-ai-security-vendors).

## FAQ

### How can I test whether a security product's score comes from a model or a rule engine?

Pull raw per-detection output for a few thousand events and count the unique score values. A model with learned parameters produces a near-continuous distribution across hundreds or thousands of distinct values. A weighted rule engine produces a small repeating set, because every score is a sum over the same fixed condition list. Twelve unique values across 8,000 events is a scoring rubric. Bucketed API output can mask a real model, so follow up by sweeping one input feature and watching whether the score steps at a threshold or bends gradually.

### What does a feature sweep reveal about a vendor's detection model?

It exposes the shape of the decision boundary. Take an event the product flags, vary one field in small increments, resubmit, and record the score. A rule fires at a fixed threshold, so the score stays flat and then jumps, usually at a round number like entropy above 3.5 or 1,000,000 bytes. A trained model bends instead: the score moves gradually, and the point where it moves shifts depending on the other fields, because the model learned an interaction between them. Get written authorization before you do this against a vendor API.

### What evidence proves a security vendor is running a maintained machine learning model?

A model version string in every detection payload (not the product release number), a retraining changelog with dates and held-out evaluation metrics per version, and a named drift monitor with a record of what happened the last time it tripped. These are expensive to fake because they require a working MLOps pipeline. A product that claims to have used ML since 2019 with no retraining history is either not retraining or not running a model.

### Is it a problem if a security tool turns out to be a rule engine rather than machine learning?

Not by itself. A well-maintained rule pack with a tight false positive budget beats a badly trained model, and for many detection problems it is the correct engineering choice. The problem is paying model prices for rules, because you get neither the generalization to unseen variants that a model provides nor the readability and predictability that make rules easy to tune. Test the claim to price the product honestly, then judge the product on detection lift over your existing stack.


---

Canonical: https://gtkcyber.com/blog/does-your-ai-security-tool-use-real-ai/