# Feature Engineering for Security Data: From Logs to ML Features

By Summer Rankin · 2026-07-01

> Feature engineering for security machine learning: encoding high-cardinality fields, building time features, and handling class imbalance in logs and flows.

## Feature engineering is where security ML succeeds or fails

Most failed security ML projects did not fail at the model. They failed at the features. Teams feed raw logs into an algorithm, get poor results, and swap classifiers hoping one is smarter. None of them are. A model only sees the numbers you hand it, and raw security data almost never separates attacker from user in its raw form.

The work is translation: turning a log line, a flow record, a URL, or an IP into features that carry signal. This is the least glamorous part of a detection project and the part that decides whether it works. If you have built a [data science workflow for incident response](/blog/data-science-for-incident-responders), you have already felt this: the answer was in the data, but not in any single raw column.

## The three problems specific to security data

**High-cardinality categorical fields.** Security data is full of them: IP addresses, user agents, domains, usernames. The instinct is one-hot encoding, which turns a field of a million distinct IPs into a million columns and kills your model. Do not do it. Instead, encode behavior. Replace the IP with derived counts:

```python
# Behavior beats identity: features per source IP
agg = df.groupby("src_ip").agg(
    distinct_dst=("dst_ip", "nunique"),
    total_bytes=("bytes", "sum"),
    conn_count=("dst_ip", "size"),
)
```

For fields you must keep, frequency encoding (replace the value with how often it appears) or target encoding keeps the signal without the column explosion. For domains, entropy and length separate algorithmically generated names from real ones, the same idea behind [detecting DGA domains in Python](/blog/detecting-dga-domains-python).

**Time features.** Timestamps are your richest source and the most wasted. Hour of day, day of week, and time since the last event from the same host all carry signal. Off-hours activity, bursts, and periodic beaconing only become visible once you compute inter-event deltas:

```python
df["timestamp"] = pd.to_datetime(df["timestamp"])
df = df.sort_values(["host", "timestamp"])

# Seconds since this host's previous event: beacons are suspiciously regular
df["delta"] = df.groupby("host")["timestamp"].diff().dt.total_seconds()
df["hour"] = df["timestamp"].dt.hour
```

A model with `delta` and `hour` can learn that a host phoning home every 60 seconds at 3am is not a person.

**Class imbalance.** This is the one that quietly ruins security models. Attacks are rare: maybe one malicious event in a hundred thousand. A model can hit 99.999 percent accuracy by calling everything benign and catching nothing. Accuracy is the wrong metric here. Track precision and recall, and address the imbalance directly:

```python
from sklearn.ensemble import RandomForestClassifier

# class_weight tells the model rare events cost more to miss
clf = RandomForestClassifier(class_weight="balanced")
```

Beyond `class_weight`, resampling the training set (undersampling the majority, or SMOTE for the minority) helps, but be honest about the base rate. The math that makes a model flood the queue with false positives is the same math behind [reducing false positives with machine learning](/blog/reducing-false-positives-security-alerts-machine-learning). Get the features and the imbalance right, and the false-positive problem mostly solves itself.

## Start with the data, not the algorithm

If you take one thing from this: spend your time on features, not on model selection. A logistic regression with good, security-aware features beats a deep network fed raw logs, every time. The teams that ship detections are the ones who understand their data well enough to build the right features from it.

We teach this hands-on, on real security datasets, in the [Applied Data Science & AI for Cybersecurity](/courses/applied-data-science-ai) course. Feature engineering gets its own extended block because it is where students see the biggest jump in results. The full course runs [at Black Hat USA 2026](/lp/applied-data-science-black-hat-2026).

The course runs as two 2-day sessions at Black Hat USA 2026, August 1 to 4 in Las Vegas: [sign up here](/lp/applied-data-science-black-hat-2026).

## FAQ

### What is feature engineering in security machine learning?

It is turning raw security data (logs, netflow, URLs, IPs) into numeric inputs a model can learn from. Raw fields like an IP address or a domain string rarely separate malicious from benign on their own, so you derive features such as entropy, counts, and ratios that do.

### How do you handle high-cardinality fields like IP addresses?

Do not one-hot encode them; you will create millions of columns. Instead derive behavior: count of distinct destinations per source, bytes-out to bytes-in ratio, or frequency of the value. Target encoding and hashing also help when you must keep the field itself.


---

Canonical: https://gtkcyber.com/blog/feature-engineering-security-machine-learning/