Machine learning in threat hunting is oversold in one specific way: vendors imply the model finds the threat. It does not. A hunt still starts with a hypothesis and a human who knows the environment. What machine learning does well is one narrow, valuable job: it shrinks the candidate set. Instead of an analyst scrolling through 40,000 outbound sessions, ML hands them 40 that do not look like the rest.
That reframing decides where ML belongs in a hunt and where it wastes your time. Here is how to apply it without pretending it replaces judgment.
Start With Whether You Have Labels
The first decision is not which algorithm to use. It is whether you have labeled examples of the thing you are hunting.
- You have labels. For well-studied problems, curated datasets exist: DGA domains, phishing URLs, known malware families. Here supervised classification works. Extract features and train a classifier. See detecting DGA domains in Python for a worked example using lexical features and a gradient-boosted model.
- You do not have labels. This is most hunting. You are looking for something you cannot name yet, so there is nothing to train a classifier against. This is where unsupervised methods earn their place: clustering to group events, and distance-based scoring to rank them.
If you are unsure which camp a hunt falls into, it is almost always the second one. For the difference in practice, see supervised vs. unsupervised learning for security.
Clustering to Collapse Candidate Sets
The most useful unsupervised technique in hunting is not anomaly scoring. It is clustering, used to reduce volume. Group thousands of similar events into a handful of clusters, then hunt the small and the odd ones.
Command-line execution (MITRE ATT&CK T1059) is a good target. Sysmon Event ID 1 and Windows Event ID 4688 give you the full command line. Most command lines in an environment are near-duplicates of each other: the same scheduled tasks, the same management scripts, the same installer strings. The rare ones are what you want.
Vectorize the command lines with a character n-gram TF-IDF, then let DBSCAN from scikit-learn label the outliers:
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import DBSCAN
# df: one row per process creation, with a 'command_line' column
vec = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 5), min_df=2)
X = vec.fit_transform(df['command_line'].fillna(''))
db = DBSCAN(eps=0.5, min_samples=5, metric='cosine').fit(X)
df['cluster'] = db.labels_
# cluster == -1 is DBSCAN's noise label: command lines that resemble
# nothing else in the dataset. Rare by construction, worth hunting.
rare = df[df['cluster'] == -1].sort_values('command_line')char_wb n-grams handle the obfuscation you see in real command lines: base64 blobs, mixed casing, inserted characters. A PowerShell -enc payload will not cluster with anything legitimate and drops straight into the noise group. The same pattern works on outbound HTTP sessions, DNS query strings, and user-agent values. You are not asking the model “is this malicious.” You are asking “is this like everything else,” which is a question ML answers reliably.
Peer-Group Anomaly for Account Behavior
Single-user baselining has a known failure mode: if an account is compromised early, or you only have a short history for it, its baseline is either poisoned or too thin to be useful. Peer-group comparison sidesteps this. Compare each account to the accounts most like it, not only to its own past.
Build a feature vector per account (distinct hosts reached, off-hours logon ratio, privileged operations, distinct source IPs), group accounts by role or organizational unit, and measure how far each account sits from its peers:
import numpy as np
# features: DataFrame indexed by account, grouped by 'role'
def peer_distance(group):
numeric = group.drop(columns='role')
centroid = numeric.mean()
std = numeric.std().replace(0, 1)
z = (numeric - centroid) / std
return np.sqrt((z ** 2).sum(axis=1)) # distance from peer centroid
features['peer_score'] = (
features.groupby('role', group_keys=False).apply(peer_distance)
)
suspects = features.sort_values('peer_score', ascending=False).head(25)A service account that suddenly behaves like an interactive admin, or a helpdesk user reaching servers no one else in helpdesk touches, surfaces here even when their own history looks unremarkable. This is the useful core of what vendors sell as UEBA, and it maps cleanly to lateral movement (MITRE ATT&CK T1021) and account manipulation hunts.
The Work Is Feature Engineering, Not the Model
Every example above spends more effort turning raw logs into features than on the algorithm. That is not incidental. The choice between DBSCAN and KMeans, or between a random forest and gradient boosting, rarely decides whether a hunt succeeds. The features do. A byte-ratio, an inter-arrival coefficient of variation, an entropy score on a domain string: these are what carry signal. Spend your time there. See feature engineering for security machine learning for the patterns that hold up.
Where Machine Learning Stops
Be honest about the limits, because the failures are predictable:
- It will not generate hypotheses. ML ranks and groups what you point it at. Deciding to hunt command-line execution, or peer-group deviation, or beaconing, is your call. The model has no idea what an attacker looks like.
- The base-rate problem is unforgiving. Malicious activity is a tiny fraction of all activity, so a small false positive rate still buries the real findings. Use ML to prioritize, then filter with cheap rules (known-good ASNs, expected admins) before an analyst sees anything.
- It is blind to living-off-the-land. If attackers use the same tools your admins use, statistically unusual is not the same as malicious. This is the same limit that constrains anomaly detection in security operations: map your MITRE ATT&CK threat model explicitly against what the model can and cannot see.
- Findings need validation before they become detections. A cluster or a high peer-score is a lead, not a verdict. Confirm it, then translate reliable logic into a production detection rather than rerunning a notebook forever.
Applied well, machine learning is a force multiplier for a hunter who already knows what questions to ask. GTK Cyber’s applied data science training covers exactly this: building and calibrating these models against realistic security datasets, with hands-on labs on the feature engineering, clustering, and peer-group techniques described here.