ML for Malware and Phishing Detection: What to Learn First

Phishing detection and malware detection get named in the same breath, then land in the same course module, and students reasonably assume the techniques transfer. They mostly do not. The classifier at the end looks similar. Everything before it is a different job.

A phishing URL is a string. You can featurize a million of them in a laptop’s memory in a few seconds, with no execution and no containment problem. A malware sample is a file that runs, and the moment you decide to featurize it you have to answer where it lives, who can touch it, and whether you trust your own parser. That difference drives everything downstream: the data you can get, how you validate, and what the model is allowed to decide.

The URL side is already written up in building an ML pipeline for phishing URL detection. This is the other half.

Static features are structure, not behavior

A Windows PE file advertises a lot about itself before it executes. Parse it with pefile or LIEF and the header alone yields a usable feature vector:

import math
import pefile
from collections import Counter

def entropy(data):
    if not data:
        return 0.0
    counts = Counter(data)
    n = len(data)
    return -sum((c / n) * math.log2(c / n) for c in counts.values())

def pe_features(path):
    pe = pefile.PE(path, fast_load=True)
    pe.parse_data_directories()
    sections = pe.sections
    imports = []
    for entry in getattr(pe, "DIRECTORY_ENTRY_IMPORT", []):
        for imp in entry.imports:
            if imp.name:
                imports.append(imp.name.decode(errors="ignore"))
    return {
        "num_sections": len(sections),
        "max_section_entropy": max((entropy(s.get_data()) for s in sections), default=0.0),
        "mean_section_entropy": sum(entropy(s.get_data()) for s in sections) / max(len(sections), 1),
        "size_of_code": pe.OPTIONAL_HEADER.SizeOfCode,
        "num_imports": len(imports),
        "has_injection_apis": int(any(
            api in imports for api in
            ("VirtualAllocEx", "WriteProcessMemory", "CreateRemoteThread")
        )),
        "timestamp": pe.FILE_HEADER.TimeDateStamp,
    }

Section entropy above roughly 7.0 says the section is compressed or encrypted, which is the signature of packing (Obfuscated Files or Information: Software Packing, T1027.002). The injection API triple maps to Process Injection (T1055). Neither is malicious on its own. Commercial software packs itself, and legitimate debuggers call WriteProcessMemory. They are features, not rules, and that distinction is the whole reason to use a model.

Start with EMBER, not with binaries

The obstacle to learning this is not the math. It is that a realistic training corpus means a large pile of live malware, and most people learning the technique should not be assembling one.

EMBER, published by Elastic, removes that problem. It ships pre-extracted 2,381-dimensional feature vectors for roughly a million PE files, labeled, with a LightGBM baseline in the repo. No executables change hands. In our courses the malware module starts here for exactly that reason: a classroom is the wrong place to distribute live samples, and the pipeline you learn on EMBER vectors is the same pipeline you later point at your own corpus.

import lightgbm as lgb

params = {
    "objective": "binary",
    "num_leaves": 2048,
    "min_data_in_leaf": 50,
    "learning_rate": 0.05,
    "feature_fraction": 0.5,
    "bagging_fraction": 0.8,
    "num_iterations": 1000,
}
model = lgb.train(params, lgb.Dataset(X_train, y_train))

Gradient boosting beats a RandomForestClassifier on this feature space by a useful margin and trains fast on a million rows. Deep learning on raw bytes (MalConv and its descendants) is worth knowing about, but it needs GPUs and buys little over boosted trees on tabular static features.

The validation mistake that makes everything look great

Never random-split malware data.

A random split scatters samples from the same family, the same campaign, and often the same build across your train and test sets. The model memorizes the family and reports 99% on the test set. Then it meets a family that shipped last week and quietly fails.

Split by time. Train on everything before a cutoff, test on everything after. EMBER is organized by month specifically so you can do this. The number you get will be lower, sometimes a lot lower, and it is the only number that predicts production behavior. Then keep measuring it: malware distributions drift faster than almost any other security dataset, so a static model degrades on a schedule you can actually plot. The same reasoning applies to any security model you intend to ship, which is the subject of evaluating ML model robustness for security use cases.

What to learn, in order

If you are building this skill deliberately, the sequence matters more than the syllabus:

1. Feature engineering on security data in pandas. Everything else is downstream of this and it is where most of the work lives.
2. Supervised classification with honest metrics. Precision and recall per class on heavily imbalanced data, never accuracy.
3. File format parsing. pefile and LIEF for PE, and the equivalent for ELF and Mach-O if your fleet needs it.
4. Temporal validation and drift measurement. The step almost every tutorial skips.
5. Adversarial machine learning. MITRE ATLAS catalogs evasion (AML.T0015) and poisoning (AML.T0020) against exactly these models.

Step five is not optional for a security audience. A malware classifier is a control that an adversary can inspect and attack. Appending bytes, padding a section, or importing a few benign-looking functions can flip a static model’s score without changing what the binary does.

What the model is allowed to decide

A classifier score is a prioritization signal. It ranks the unknown remainder after your signatures and reputation feeds have done their work, so an analyst opens the right file first.

It is not a verdict, and it does not replace reverse engineering. When the question is what this sample does, who sent it, and what it touched, someone still opens it in a disassembler. The model shortens the queue; it does not answer the question. Teams that wire a raw classifier output straight into a blocking decision discover their false positive rate on packed internal tooling and legitimate installers the hard way.

Both halves of this, the URL classifier and the file classifier, are labs in GTK Cyber’s Applied Data Science and AI for Cybersecurity and Threat Hunting with Data Science courses, built on real data with the temporal-split discipline baked in rather than bolted on at the end.