# Getting Started with Pandas for Security Data Analysis

By Charles Givre · 2026-06-29

> Why Pandas beats grep and Excel for security data analysis, with concrete operations for loading logs, filtering, and building time-based features.

## Why Pandas beats grep and Excel

grep is great for finding a string. It is bad at answering "which accounts had more than 20 failed logins in a five-minute window." Excel can do that, until your log file has three million rows and the app gives up.

Pandas sits in the middle: it loads structured data into memory, lets you filter and aggregate with a few lines, and keeps every step as code you can rerun next week. For security data, where the same auth log question comes up constantly, that repeatability is the real win. Your analysis becomes a script, not a spreadsheet you cannot reproduce.

## A few operations that cover most of the work

Loading a log is one line. Most security logs are CSV or JSON lines:

```python
import pandas as pd

# Load an auth log exported as CSV
df = pd.read_csv("auth.log.csv")

# For big files, load only the columns you need
df = pd.read_csv("auth.log.csv", usecols=["timestamp", "user", "src_ip", "result"])
```

Filtering is how you narrow to the interesting subset. Here, just the failed logins:

```python
# Boolean filtering: keep only failures
failures = df[df["result"] == "failed"]
```

The operation that earns its keep is `groupby`. Password spraying, brute force, and unusual access patterns all show up as counts per key:

```python
# Count failed logins per source IP, sorted worst-first
spray = (
    failures.groupby("src_ip")
    .size()
    .sort_values(ascending=False)
)
```

That is three lines to surface the spraying source that a dashboard buried under noise. This is the same first move behind almost every [threat hunting pipeline in Python and Jupyter](/blog/threat-hunting-pipeline-python-jupyter): load, filter, group, look.

Time is where security data gets interesting, so parse timestamps early and build features from them:

```python
# Parse the timestamp column into real datetimes
df["timestamp"] = pd.to_datetime(df["timestamp"])

# Derive hour-of-day: 3am logins are worth a second look
df["hour"] = df["timestamp"].dt.hour

# Failed logins per user per hour
per_hour = (
    failures.set_index("timestamp")
    .groupby("user")
    .resample("1h")
    .size()
)
```

Hour-of-day and per-window counts are the raw material for detection. A human logging in is boring. The same account logging in at 3am from a new IP, 40 times an hour, is not.

## Where analysts get stuck

Three things trip people up. First, timestamps: if you skip `pd.to_datetime`, every time-based operation silently treats your timestamps as strings and gives nonsense. Parse them first, always.

Second, chained filtering and the `SettingWithCopyWarning`. When you filter and then assign a new column, Pandas warns you may be editing a copy. Use `.copy()` when you intend to keep a filtered subset, and the warning goes away along with the subtle bugs.

Third, memory. A multi-gigabyte log will not fit at once. Use `chunksize` in `read_csv` to stream it, or filter columns with `usecols` so you load only what matters. Analysts abandon Pandas here when the fix is one parameter.

None of this is hard. It just needs reps on real data, which is exactly what we do in the [Applied Data Science & AI for Cybersecurity](/courses/applied-data-science-ai) course. We start in Pandas on security datasets before touching a single model, and we run the full course [at Black Hat USA 2026](/lp/applied-data-science-black-hat-2026).

The course runs as two 2-day sessions at Black Hat USA 2026, August 1 to 4 in Las Vegas: [claim your seat](/lp/applied-data-science-black-hat-2026).

## FAQ

### Why use Pandas instead of Excel for log analysis?

Excel caps out around a million rows and slows to a crawl well before that. Pandas handles tens of millions of rows in memory, keeps your steps in reusable code, and connects directly to scikit-learn. You also get a repeatable analysis instead of a spreadsheet nobody can reproduce.

### How do you load a large log file into Pandas?

Use pd.read_csv for delimited logs, or pd.read_json for JSON lines. For files too big for memory, pass chunksize to read_csv and process in batches, or filter columns with usecols so you only load what you need.


---

Canonical: https://gtkcyber.com/blog/pandas-for-security-data-analysis/