# How to Apply Machine Learning to Threat Hunting

By Charles Givre · 2026-07-10

> Machine learning won't tell you what to hunt. It collapses huge candidate sets into something a human can review. Here's where clustering, classification, and peer-group models fit a real hunt.

Machine learning in threat hunting is oversold in one specific way: vendors imply the model finds the threat. It does not. A hunt still starts with a hypothesis and a human who knows the environment. What machine learning does well is one narrow, valuable job: it shrinks the candidate set. Instead of an analyst scrolling through 40,000 outbound sessions, ML hands them 40 that do not look like the rest.

That reframing decides where ML belongs in a hunt and where it wastes your time. Here is how to apply it without pretending it replaces judgment.

## Start With Whether You Have Labels

The first decision is not which algorithm to use. It is whether you have labeled examples of the thing you are hunting.

- **You have labels.** For well-studied problems, curated datasets exist: DGA domains, phishing URLs, known malware families. Here supervised classification works. Extract features and train a classifier. See [detecting DGA domains in Python](/blog/detecting-dga-domains-python) for a worked example using lexical features and a gradient-boosted model.
- **You do not have labels.** This is most hunting. You are looking for something you cannot name yet, so there is nothing to train a classifier against. This is where unsupervised methods earn their place: clustering to group events, and distance-based scoring to rank them.

If you are unsure which camp a hunt falls into, it is almost always the second one. For the difference in practice, see [supervised vs. unsupervised learning for security](/blog/supervised-vs-unsupervised-learning-security).

## Clustering to Collapse Candidate Sets

The most useful unsupervised technique in hunting is not anomaly scoring. It is clustering, used to reduce volume. Group thousands of similar events into a handful of clusters, then hunt the small and the odd ones.

Command-line execution (MITRE ATT&CK [T1059](https://attack.mitre.org/techniques/T1059/)) is a good target. Sysmon Event ID 1 and Windows Event ID 4688 give you the full command line. Most command lines in an environment are near-duplicates of each other: the same scheduled tasks, the same management scripts, the same installer strings. The rare ones are what you want.

Vectorize the command lines with a character n-gram TF-IDF, then let [`DBSCAN`](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html) from [scikit-learn](https://scikit-learn.org/) label the outliers:

```python
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import DBSCAN

# df: one row per process creation, with a 'command_line' column
vec = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 5), min_df=2)
X = vec.fit_transform(df['command_line'].fillna(''))

db = DBSCAN(eps=0.5, min_samples=5, metric='cosine').fit(X)
df['cluster'] = db.labels_

# cluster == -1 is DBSCAN's noise label: command lines that resemble
# nothing else in the dataset. Rare by construction, worth hunting.
rare = df[df['cluster'] == -1].sort_values('command_line')
```

`char_wb` n-grams handle the obfuscation you see in real command lines: base64 blobs, mixed casing, inserted characters. A `PowerShell -enc` payload will not cluster with anything legitimate and drops straight into the noise group. The same pattern works on outbound HTTP sessions, DNS query strings, and user-agent values. You are not asking the model "is this malicious." You are asking "is this like everything else," which is a question ML answers reliably.

## Peer-Group Anomaly for Account Behavior

Single-user baselining has a known failure mode: if an account is compromised early, or you only have a short history for it, its baseline is either poisoned or too thin to be useful. Peer-group comparison sidesteps this. Compare each account to the accounts most like it, not only to its own past.

Build a feature vector per account (distinct hosts reached, off-hours logon ratio, privileged operations, distinct source IPs), group accounts by role or organizational unit, and measure how far each account sits from its peers:

```python
import numpy as np

# features: DataFrame indexed by account, grouped by 'role'
def peer_distance(group):
    numeric = group.drop(columns='role')
    centroid = numeric.mean()
    std = numeric.std().replace(0, 1)
    z = (numeric - centroid) / std
    return np.sqrt((z ** 2).sum(axis=1))   # distance from peer centroid

features['peer_score'] = (
    features.groupby('role', group_keys=False).apply(peer_distance)
)
suspects = features.sort_values('peer_score', ascending=False).head(25)
```

A service account that suddenly behaves like an interactive admin, or a helpdesk user reaching servers no one else in helpdesk touches, surfaces here even when their own history looks unremarkable. This is the useful core of what vendors sell as UEBA, and it maps cleanly to lateral movement (MITRE ATT&CK [T1021](https://attack.mitre.org/techniques/T1021/)) and account manipulation hunts.

## The Work Is Feature Engineering, Not the Model

Every example above spends more effort turning raw logs into features than on the algorithm. That is not incidental. The choice between `DBSCAN` and `KMeans`, or between a random forest and gradient boosting, rarely decides whether a hunt succeeds. The features do. A byte-ratio, an inter-arrival coefficient of variation, an entropy score on a domain string: these are what carry signal. Spend your time there. See [feature engineering for security machine learning](/blog/feature-engineering-security-machine-learning) for the patterns that hold up.

## Where Machine Learning Stops

Be honest about the limits, because the failures are predictable:

- **It will not generate hypotheses.** ML ranks and groups what you point it at. Deciding to hunt command-line execution, or peer-group deviation, or beaconing, is your call. The model has no idea what an attacker looks like.
- **The base-rate problem is unforgiving.** Malicious activity is a tiny fraction of all activity, so a small false positive rate still buries the real findings. Use ML to prioritize, then filter with cheap rules (known-good ASNs, expected admins) before an analyst sees anything.
- **It is blind to living-off-the-land.** If attackers use the same tools your admins use, statistically unusual is not the same as malicious. This is the same limit that constrains [anomaly detection in security operations](/blog/anomaly-detection-security-operations): map your MITRE ATT&CK threat model explicitly against what the model can and cannot see.
- **Findings need validation before they become detections.** A cluster or a high peer-score is a lead, not a verdict. Confirm it, then translate reliable logic into a production detection rather than rerunning a notebook forever.

Applied well, machine learning is a force multiplier for a hunter who already knows what questions to ask. GTK Cyber's applied data science training covers exactly this: building and calibrating these models against realistic security datasets, with hands-on labs on the feature engineering, clustering, and peer-group techniques described here.

## FAQ

### Should I use supervised or unsupervised machine learning for threat hunting?

It depends on whether you have labels. Supervised models need examples tagged malicious or benign, which you rarely have for a novel hunt but do have for well-studied problems like DGA domains or phishing URLs. Most hunting is unsupervised because you are looking for something you cannot yet name. Unsupervised methods (clustering, peer-group distance, anomaly scoring) group or rank events so a human reviews tens of candidates instead of tens of thousands. Start unsupervised for exploratory hunts and reach for supervised classification only when you have a labeled dataset that reflects your environment.

### How can clustering help with threat hunting?

Clustering collapses a large volume of similar events into a small number of groups so you hunt the groups instead of the raw events. Vectorize process command lines with a character n-gram TF-IDF and run DBSCAN with a cosine metric: the points DBSCAN labels as noise (cluster -1) are the rare, one-off command lines that do not resemble anything else in the dataset. Those are the ones worth a closer look. The same idea applies to outbound network sessions, DNS query patterns, and user-agent strings.

### What is peer-group anomaly detection and why is it better than per-user baselining?

Peer-group anomaly detection compares an account to the accounts most similar to it (same department, role, or OU) rather than only to its own history. A compromised account that starts touching new systems may look normal against its own baseline if that baseline is short or already poisoned, but it stands out against its peers who do not do those things. Compute each account's feature vector, group by role, and measure distance from the group centroid or from k nearest neighbors within the group. This catches first-time-abused accounts that single-user baselines miss.

### Why does machine learning generate so many false positives in threat hunting?

Because malicious activity is extremely rare relative to normal activity, so even a low false positive rate produces many more false alarms than true detections. This is the base-rate problem. A model with a 1% false positive rate on 1,000,000 daily events flags 10,000 events, and the handful of real ones are buried. The fix is not a better model alone: use ML to rank and cluster, then apply cheap rule-based filters (known-good ASNs, expected admin accounts, business-hours context) before anything reaches an analyst.


---

Canonical: https://gtkcyber.com/blog/machine-learning-for-threat-hunting/