# Data Science for Managers: What to Ask Your Team

By Charles Givre · 2026-08-07

> Data science for managers in security: why accuracy is the wrong metric, how to spot leakage in a model review, and what to ask before funding a project.

A model that never fires can be 99.99% accurate on your security data. That single fact is the reason most managers cannot evaluate the work their data science team brings them.

Run the arithmetic. One million events a day, one hundred of them malicious. A classifier that labels everything benign gets 999,900 of a million right. It catches nothing, and it beats the accuracy number on most vendor slides.

## The number that should end most model reviews

Now take a real detector: 99% recall, and a false positive rate of one in a thousand. It catches 99 of the 100 attacks, which sounds excellent, and it also fires on roughly 1,000 of the 999,900 benign events. Your analysts open 1,099 alerts to find 99 real ones. Precision is 9%.

Nothing is broken in that model. The math is doing what it should. What changed is the question a manager should be asking, from "how accurate is it" to "what does the queue look like on Monday." Those are different questions and only the second one has a budget attached.

So when a team presents a result, ask for precision and recall at the threshold they intend to deploy at. Not the best threshold on the curve. The one going to production. If the answer is a single accuracy figure, the review is not finished, and [scikit-learn's precision-recall documentation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html) is a reasonable thing to send back with the request.

## Ask what the baseline is

The most useful question in a project review is also the least popular: what does the dumb version get?

Before a model is worth funding, someone should have measured a heuristic on the same data. A threshold on connection duration. A regex on command lines. A list of the twenty riskiest processes. Often the heuristic gets 80% of the benefit for two days of work, and the honest conclusion is to ship the rule and stop.

This is not an anti-modeling position. It is the first rule in Google's [Rules of Machine Learning](https://developers.google.com/machine-learning/guides/rules-of-ml): do not be afraid to launch without machine learning. A team that has never established a baseline cannot tell you how much of their result came from the model rather than from finally cleaning up the data, and the cleaning is usually where the gain came from.

## Ask how the data was split

This is the failure I see most often, and it is invisible on the slide.

Security data is time-ordered. If a team splits it randomly into training and test sets, the model trains on Thursday and gets tested on Wednesday, which means it has seen the future. Accuracy looks wonderful in the notebook and collapses in production. Same effect if a feature quietly encodes the answer: a field populated by the incident response process, an enrichment applied only to events someone already investigated.

The question to ask is plain. Was the test set later in time than the training set? [`TimeSeriesSplit`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html) is the standard way to do it and takes one line, so there is no cost excuse. A team that split by time and reports a lower number is giving you a more valuable result than a team that split randomly and reports a higher one.

## Managing work that does not have a deliverable

Software projects converge. Data science projects sometimes conclude that the thing is not predictable from the data you have, and that is a legitimate outcome rather than a failure.

Budget accordingly: two to four weeks of discovery and data exploration, another two to six to a first working model, then iteration with no natural end. Write the kill criteria at the start, while everyone is still calm, because deciding to stop is much harder after four months of sunk effort. And expect the first production model to be wrong, since it will meet traffic the training data never contained.

The management error is treating exploration as a phase to compress. It is where you learn whether the project is possible.

## Most of it is not modeling

Sculley and colleagues made this point in their 2015 paper on hidden technical debt in machine learning systems: the model is a small box in the middle of a much larger diagram. Data collection, feature plumbing, monitoring, and serving are the rest, and they are where the maintenance cost lives.

Practically, that means resourcing a data engineer before a second data scientist on most security teams, and it means the monitoring question ("how will we know when this model stops working?") belongs in the project plan rather than in a follow-up. Start lean on infrastructure and add it when a specific bottleneck appears. Teams that buy the platform first tend to spend a year configuring it.

## Where this advice runs out

None of this substitutes for technical review. A manager who can read a notebook and ask the four questions above will catch inflated results and bad framing. Catching a subtle bug in a feature transformation takes someone who does this for a living, so if the model is going to drive a consequential decision, get a second practitioner to review it.

It also assumes you have the data. If the events you need are not being collected, or are sampled, or land in three systems with no common key, no amount of project management fixes it. That is an engineering problem wearing a data science hat, and it should be scoped as one.

Reading your team's work is a learnable skill, which is why the [Data Science for Managers](/courses/data-science-for-managers) course we teach splits its two days evenly between instruction and hands-on exercises: managers leave able to open the notebook, not just the summary. For the practitioner side of the same material, see [the data science skills SOC analysts need](/blog/data-science-skills-soc-analysts-2026) and [how to cut false positives with machine learning](/blog/reducing-false-positives-security-alerts-machine-learning).

## FAQ

### Why is accuracy a bad metric for security machine learning?

Because security data is extremely imbalanced. If one million events a day contain one hundred malicious ones, a model that labels every single event benign is correct 999,900 times out of a million, which is 99.99% accuracy and zero detections. Accuracy rewards predicting the majority class, and in security the majority class is 'nothing happened.' Ask for precision and recall at the operating threshold instead, or the precision-recall curve across thresholds. Those metrics describe what the analyst queue will actually look like.

### What should a manager ask in a data science model review?

Four questions cover most of it. What is the baseline, meaning what does a simple rule or heuristic achieve on the same data? How was the data split into training and test sets, and was the split done by time? What are precision and recall at the threshold you plan to deploy at, not at the best threshold? And what happens when the model is wrong, meaning who absorbs the false positives and what does a miss cost? A team that cannot answer these has not finished the work.

### How long should a data science project take?

Plan roughly two to four weeks for discovery and data exploration, then another two to six weeks to a first working model, and treat deployment, monitoring, and iteration as ongoing rather than a phase with an end date. The important management adjustment is that the first model will be wrong in production and the budget has to include the iteration. Any proposal promising a production-ready detection model in two weeks is either reusing something that already exists or overselling.

### Do managers need to write code to oversee a data science team?

Not production code, but you need enough fluency to read your team's work. A manager who can open a notebook, follow how the features were built, and see which rows went into the test set can evaluate a claim. A manager who cannot is dependent on the summary slide, and the summary slide is where the leakage and the threshold games hide. That is the reason our managers course spends half its class time on hands-on exercises rather than lecture.

### Should I hire a PhD data scientist for my security team?

Usually not as a requirement. PhD training pays off for original research and novel modeling. Most applied security data science is well served by strong programming, solid statistics, and business judgment, which shrinking the candidate pool to PhDs filters out for no gain. A more useful hiring signal is whether the candidate reaches for the simplest model that solves the problem. Someone who proposes deep learning for a task a logistic regression handles will make the same call on your budget.


---

Canonical: https://gtkcyber.com/blog/data-science-for-managers/