scikit-learn shows up in most security ML tutorials as one thing: an anomaly detector. IsolationForest, a contamination parameter, done. That is a fraction of what the library does for log data. Two other jobs matter more day to day in a SOC: grouping tens of thousands of near-identical log lines so an analyst reviews ten clusters instead of ten thousand events, and classifying events so known-benign noise stops paging anyone.
This post covers those two workflows. It assumes you can already load a log into a DataFrame. If not, start with Pandas for security data analysis and come back.
From log lines to a feature matrix
scikit-learn models take a numeric matrix, not raw log text. Building that matrix is most of the work. Two encoders cover the majority of log fields:
- Structured fields (port, bytes, status code, hour of day) go in as numbers, scaled with
StandardScalerso no single large-magnitude column dominates. - Free-text fields (URLs, process command lines, user agents, syslog messages) go through
TfidfVectorizer, which turns text into weighted token vectors.
For a proxy log, that looks like this:
from sklearn.feature_extraction.text import TfidfVectorizer
# df['url'] is the requested URL per row
vec = TfidfVectorizer(analyzer='char_wb', ngram_range=(3, 5), min_df=5)
X = vec.fit_transform(df['url'])Character n-grams (char_wb, 3 to 5 characters) work better than word tokens on URLs and command lines, where the signal lives in substrings like /wp-admin or a base64 chunk, not in whitespace-separated words.
Workflow 1: cluster to triage volume
A single web server can emit tens of thousands of near-identical log lines an hour. Clustering collapses them. MiniBatchKMeans scales to large logs and groups the TF-IDF vectors:
from sklearn.cluster import MiniBatchKMeans
km = MiniBatchKMeans(n_clusters=50, random_state=42)
df['cluster'] = km.fit_predict(X)
# Rare shapes hide in the smallest clusters
sizes = df['cluster'].value_counts()The move that pays off is sorting clusters by size and reviewing the smallest ones first. The giant clusters are your normal traffic. The cluster of 12 requests that resembles none of the other 49 clusters is the one worth an analyst’s time. DBSCAN is an alternative when you do not want to pick n_clusters up front, at the cost of tuning eps.
This is triage, not detection. Clustering tells you what is unusual in shape, not what is malicious. An analyst still reads the small clusters and makes the call.
Workflow 2: classify known event types
Once you have labels (from past investigations, a SIEM’s verdicts, or a public corpus), a supervised classifier can carry the repetitive decision. A RandomForestClassifier is a strong default on the mixed numeric-and-text features above:
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
X_train, X_test, y_train, y_test = train_test_split(
X, df['label'], test_size=0.2, stratify=df['label'], random_state=42)
clf = RandomForestClassifier(n_estimators=300, class_weight='balanced', n_jobs=-1)
clf.fit(X_train, y_train)
print(classification_report(y_test, clf.predict(X_test)))Two arguments earn their place. class_weight='balanced' stops the model from predicting “benign” for everything when 99.9% of your logs are benign. stratify=df['label'] keeps the rare class present in both the train and test splits.
The public Loghub collection from the LogPAI group is a good place to practice. It has labeled system logs (HDFS, BGL, and others) with anomaly labels, so you can build and test a classifier without waiting for your own labeled incidents to pile up.
Read the right metric
Accuracy lies on security data. A classifier that calls everything benign scores 99.9% accuracy on a log where 1 in 1,000 events is malicious, and it catches nothing. Read precision and recall per class from classification_report, and pick the tradeoff deliberately: a SOC drowning in alerts wants precision, a hunt for a known-bad pattern wants recall. For scoring outliers rather than known classes, that is the anomaly detection job, which uses a different family of models.
The mistake we see most often when we teach this is students optimizing accuracy and declaring victory. On imbalanced security data that number is meaningless. We spend real lab time in the Applied Data Science and AI for Cybersecurity course on reading a confusion matrix and choosing the metric that matches the mission, because it is the difference between a model that ships and one that quietly misses everything.
Where scikit-learn stops
scikit-learn is a batch library. It fits and predicts on data held in memory; it is not a streaming engine. For real-time scoring at SOC scale, you train in scikit-learn and then serve the fitted model behind your pipeline (a Kafka consumer, a Spark job, or a detection rule that calls the serialized model). Every model here also decays as traffic shifts: a classifier trained on last quarter’s URLs slowly goes stale. Plan to retrain, and track precision and recall over time so you notice the drift before your analysts do.
If you want reps on this with real security datasets, and an instructor who can tell you why a model did something surprising, that is what the Applied Data Science course at Black Hat USA 2026 is built for.