Teams new to machine learning in security tend to ask the wrong first question. They ask “which algorithm should I use,” when the question that actually decides everything is “do I have labels.” That single fact splits nearly every security ML problem into supervised or unsupervised, and getting the split right saves you weeks of wasted effort.
Supervised: You Have Labels, and You Trust Them
Supervised learning trains on examples where the answer is known. You feed the model thousands of samples tagged malware or benign, phishing or legitimate, and it learns the boundary between them. At inference time it assigns new samples to one of those known classes.
Supervised methods shine when three things are true: the categories are well defined, you have enough labeled examples of each, and the labels are accurate. Security has several problems that fit this cleanly.
- Malware classification. Given a corpus of confirmed malicious and benign binaries, a classifier learns byte patterns, import tables, and section entropy that separate the two.
- Phishing detection. Labeled URLs and email bodies train a model on lexical and structural features. See our walkthrough of building an ML phishing detection pipeline for a concrete end-to-end example.
- Spam and abuse filtering. Decades of labeled data make this one of the most mature supervised problems in security.
The catch is label quality. A supervised model is only as good as its training data, and security labels are noisy. Mislabeled samples, class imbalance (far more benign than malicious), and concept drift (attackers change tactics) all degrade a model that looked great in the lab. Supervised learning does not free you from analysis. It moves your effort into building and maintaining a trustworthy labeled set.
Unsupervised: No Labels, Find the Structure
Unsupervised learning has no answer key. It models the data as it is and surfaces structure: groups of similar events (clustering) or points that do not fit the bulk of the data (anomaly detection).
This is the right tool when you do not know in advance what you are looking for. Hunting is the obvious case.
- Clustering login sessions or process trees groups similar behavior so an analyst reviews one representative per cluster instead of thousands of raw events.
- Anomaly detection flags the authentication event, the outbound flow, or the rare parent-child process relationship that sits far from normal. Our post on how anomaly detection works in security ops covers the math and the tuning traps.
- Beaconing detection uses periodicity and volume patterns to isolate command-and-control traffic without a signature.
Unsupervised methods find novel patterns that no signature and no classifier were built to catch. The tradeoff is interpretation. The model tells you something is unusual, not that it is malicious. Unusual and bad are not the same thing, and a large share of anomalies are benign rare events: a new admin tool, a quarterly batch job, a misconfigured device. You pay for the flexibility with triage.
Most Production Security ML Uses Both
The supervised-versus-unsupervised framing is useful for choosing a starting point, but real detection stacks rarely pick one. They chain them.
A typical arrangement runs unsupervised anomaly detection wide to cut volume, then hands survivors to a supervised classifier that scores them against known-bad categories. Anomalies that analysts confirm as a repeatable threat become labeled data, which trains the next supervised model. The unsupervised layer keeps finding the new, the supervised layer keeps getting sharper on the known, and the two feed each other over time.
That feedback loop is the actual point. Treat the two families as stages in one workflow rather than competing choices, and you build detection that improves instead of one that ages out.
If you want hands-on practice building both, our Applied Data Science and AI for Cybersecurity course walks through supervised classifiers and unsupervised anomaly detection on real security data, and it runs live at Applied Data Science at Black Hat USA 2026.
Ready to build detection that gets smarter over time? Reserve your seat at Black Hat USA 2026.