Fitting an isolation forest takes four lines of Python. Evaluating one honestly takes labeled attack data, a defensible unit of analysis, and arithmetic that most training on ML-based anomaly detection never gets around to.
That imbalance is the problem. Model fitting gets the lab time because it demos well and finishes fast. Evaluation gets a slide about precision and recall. Then the model reaches production, the queue fills with executives and backup service accounts, and the team concludes that ML does not work on their data.
The scoring mechanics are already written up here: how anomaly detection works in security ops covers the model families, and applying anomaly detection to authentication logs covers per-account baselines. This is about what a curriculum has to teach around them.
Start with the base rate, not the algorithm
The first exercise should be arithmetic on paper, before anyone imports scikit-learn.
Los Alamos National Laboratory released 58 days of authentication and network records from its internal enterprise network: roughly 1.05 billion authentication events across 12,425 users, with 749 events labeled as red team activity. The base rate of malicious authentication is about seven in ten million.
Run a detector over that at a false positive rate of 0.1%, which sounds like a good number in a vendor briefing. You get about 1,050,000 false positives, or 18,000 alerts a day. Catch 90% of the red team and you get 674 true positives. Precision is 0.064%: one real finding per 1,560 alerts.
To reach a precision near 40% on that data, the detector needs a false positive rate around one in a million. Three orders of magnitude better than the figure on the slide.
Stefan Axelsson published this argument in 2000 in The Base-Rate Fallacy and the Difficulty of Intrusion Detection (ACM TISSEC 3(3)). The models have changed since. The arithmetic has not, and a course that skips it produces graduates who tune toward recall and are surprised by the queue.
The unit of analysis buys more than the algorithm does
There is a cheaper lever than model quality, and it is the row definition.
Score raw events and the denominator is 1.05 billion. Aggregate the same data into user-days and it is 12,425 times 58, about 720,650 rows. At an unchanged 0.1% false positive rate that is roughly 721 false positives across the whole 58 days, around 12 a day, while the 749 red team events collapse into a much smaller number of compromised user-days. Same model, same false positive rate, a detection that a two-person team can actually work.
The cost is real: you lose event-level localization, and short-lived activity that starts and finishes inside one bucket stops standing out. Choosing the bucket is a modeling decision with a precision consequence, which is why it belongs in the syllabus next to feature engineering rather than in a footnote about data prep.
Teach precision at k, because that is what the SOC feels
Whatever the course reports, it should not be accuracy, and it should not be ROC AUC alone. Both are insensitive to the base rate that dominates the result.
Pick k from analyst capacity, then measure:
import numpy as np
from sklearn.metrics import average_precision_score
def precision_at_k(scores, labels, k):
# scores from score_samples(): lower is more anomalous
top = np.argsort(scores)[:k]
return labels[top].sum() / k
for k in (10, 40, 100, 500):
p = precision_at_k(scores, labels, k)
print(f"k={k:4d} precision={p:.3f} found={int(p * k)}")
print("average precision:", average_precision_score(labels, -scores))average_precision_score summarizes the precision-recall curve and moves when the base rate moves, which is the behavior you want from a headline metric here. The per-k table is what you show the SOC manager, because it answers the only question they asked: if my analysts work 40 of these a day, how many are real?
Fit on an earlier window and score forward. Fitting and evaluating on the same window inflates every number in that table, for the same reason a random split inflates a malware classifier, which we covered in ML for malware and phishing detection.
Four questions to ask before you pay for a course
- Does it use labeled attack data, or injected synthetic outliers? Synthetic outliers teach the API. They cannot produce an honest precision number.
- Do the labs make you tune to an alert budget? If the exercise ends at
fit_predict, the hard half was skipped. - Is the unit of analysis a decision students make, or a shape the notebook hands them? The second is a demo.
- Does it cover drift measurement? A detector that was calibrated in March and unmonitored in June is an unmeasured control.
When this training is the wrong purchase
Two situations, and both are common enough to name.
If the team cannot write basic Python, an anomaly detection course is premature. Our Threat Hunting with Data Science course lists basic Python as its prerequisite and points people without it to Python for Security Analysts first, because four days is not enough to teach both a language and a modeling discipline.
If the organization cannot produce 30 days of centralized authentication or network telemetry, the modeling skills have nowhere to land. Fix retention and collection first. Anomaly detection is a technique for teams that already have the data and cannot read all of it, not a substitute for having the data.
Also worth saying plainly: this training does not replace rules or threat intelligence. It covers the unlabeled remainder after those have done their work.
We teach the anomaly detection block of that course with half of class time in Jupyter labs, and model tuning to reduce false positives and organization-specific model creation are two of its eight listed topics for the reason above: the precision arithmetic, not the model call, is where the detection gets built.