A fraud model is the only detection system in a bank that the adversary gets to query all day, at will, with a clean answer on every attempt. Approve or decline is a label. Card testing with a run of small transactions is not reconnaissance in any loose sense; it is a labeled query campaign against a classifier, and it is how the attacker learns the decision boundary without ever seeing the model.
Financial institutions have more production ML in the path of real money than almost any other sector, and the security teams asked to defend it were trained on networks, endpoints, and web applications. The gap is narrow and specific, and closing it does not require a data science curriculum.
Query Access Is the Exposure
Adversaries do not need gradients or weights to attack a deployed classifier. Black-box optimization (MITRE ATLAS AML.T0043.001) reconstructs enough of a decision boundary from output labels alone to find inputs that cross it, and the goal, evading the model (ATLAS AML.T0015), needs nothing else.
What limits the attack is query budget. Every probe costs the adversary a transaction, a card, or an account. Which reframes controls you already own: velocity limits, device reputation, and account-age gates are not only fraud controls, they are rate limits on the adversary’s learning loop. A team that understands the model as a queryable oracle will argue for those limits differently than a team that treats the model as a black box the vendor tuned.
Constrain the Attack or the Number Is Fiction
Here is where most first attempts go wrong. Adversarial ML libraries were built for images, where any pixel can take any value and an unconstrained perturbation is still a picture. Tabular financial features do not work that way. Turn a generic attack loose on a transaction record and it returns an evasive example with a negative transfer amount, an account age that decreased since last month, and a device first seen next Tuesday. The attack succeeded against the model and describes nothing an adversary can do.
Write down the action space first: per feature, does the adversary control it, in which direction, and at what cost?
import numpy as np
FEATURES = ["amount", "hour_of_day", "device_age_days", "velocity_1h", "acct_age_days"]
# Attacker-reachable moves, in scaled feature units, with the sign of the
# available direction. acct_age_days is absent because it cannot be moved.
ACTIONS = {
"amount": (-0.9, 0.0), # smaller transfers are always available
"hour_of_day": (-8.0, 8.0), # free
"device_age_days": (0.0, 30.0), # only increases, and only by waiting
"velocity_1h": (-4.0, 0.0), # slowing down is free, speeding up is not
}
def cost_to_evade(model, x, trials=500, rng=np.random.default_rng(0)):
"""Fewest attacker actions that turn a decline into an approve."""
cheapest = None
for _ in range(trials):
cand, moves = x.copy(), 0
for feat, (lo, hi) in ACTIONS.items():
step = rng.uniform(lo, hi)
if abs(step) < 0.05:
continue
cand[FEATURES.index(feat)] += step
moves += 1
if model.predict(cand.reshape(1, -1))[0] == 0: # scored legitimate
cheapest = moves if cheapest is None else min(cheapest, moves)
return cheapest # None = no evasion foundRun that over held-out confirmed-fraud records and you get a distribution instead of a score. If 60 percent of declined transactions become approvals after two reachable moves, the model’s AUC is not the number that describes your risk. Cost-to-evade is, and it is denominated in things a fraud team already reasons about: attempts, cards, waiting time.
Use Adversarial Robustness Toolbox for the real version. HopSkipJump is decision-based, so it works against a model that returns only approve or decline, which is the access an external adversary actually has. Apply the same feature mask you defined above, because ART will otherwise perturb whatever you hand it.
The Governance Hook Already Exists
Security teams in banks tend to pitch this work as new risk requiring new budget. It is easier than that. SR 11-7, the 2011 Federal Reserve and OCC guidance on model risk management, already requires effective challenge and ongoing monitoring for models in use. Adversarial robustness is validation evidence under that standard: performance on adversary-chosen inputs rather than on inputs the historical sample happened to contain.
That makes the deliverable format the decision that matters. An evasion writeup filed as a red-team finding gets queued behind everything else in the security backlog. The same result filed as a validation finding against a model identifier in the model inventory has a remediation owner, a due date, and a validator who is obligated to look at it.
On the European side, DORA Article 26 requires threat-led penetration testing every three years for identified entities, scoped to systems supporting critical or important functions. It never says “model,” which is why the scoping conversation is worth having early: if payment fraud scoring supports a critical function, the classifier is in scope, and a test of the API in front of it is not a test of it.
Where This Training Does Not Pay Off
If your security team cannot get either the production model, a surrogate, or a scoring endpoint, none of the above runs, and in plenty of institutions that access takes longer to negotiate than the training takes to deliver. Start the request first.
When model access is genuinely blocked, test the pipeline instead, which is often the softer target anyway. Fraud and AML models retrain on analyst dispositions, so the label feedback loop is a poisoning path (ATLAS AML.T0020): an adversary who can influence which cases get marked legitimate, through mule accounts that generate clean history or by exhausting a review queue, is editing next month’s training set. That attack needs no model access at all.
And treat a failed evasion search as a weak result rather than a clean bill of health. A random search inside the action space gives a floor on the attacker’s cost, not a bound. Finding nothing means your search was not strong enough, which is exactly the honest sentence to put in the validation writeup.
We teach evasion, poisoning, and model extraction as labs rather than lecture in Applied Data Science and AI for Cybersecurity, half of which is hands-on notebook work, and financial services teams usually want the fraud-model version of those labs against their own feature set, which is what a custom engagement is for. Details on delivery inside a regulated environment are on the financial services training page, and the model-agnostic methodology is in how to evaluate ML model robustness for security use cases.