Why Accuracy Is the Wrong Metric for Fraud Detection

Your fraud model scores 99.9% accuracy on the test set. That number is real, defensible, and completely useless.

Not because the model is bad. Because accuracy, as a metric, is structurally incapable of telling you whether a fraud model works. The reason has nothing to do with the model. It has to do with the data.

The math that hides the failure

A typical credit card portfolio processes about 30 million transactions per month. Of those, roughly 18,000 are fraudulent. That's a fraud rate of 0.06%, or 6 basis points.

Build a model that labels every single transaction as legitimate. No machine learning, just a single line of code that returns "not fraud" regardless of input. That model is 99.94% accurate. It also catches exactly zero fraud.

This is the class imbalance problem, and it makes accuracy meaningless for any task where one outcome is rare. When 99.94% of your labels are the same class, a model can score near-perfect accuracy by learning nothing at all. It just bets on the majority class and wins almost every time.

Your 99.9% accurate model might be catching fraud. It might also be a glorified coin flip that happens to agree with the base rate. Accuracy can't tell you which.

What you actually need to measure

Fraud detection is a search problem. You have 30 million transactions. Somewhere in that pile are 18,000 needles. The question your model answers is: which transactions should a human investigator look at?

That question has two failure modes, and they cost different amounts.

Missing real fraud (false negatives): A fraudulent transaction slips through. The cardholder disputes it. You eat the chargeback, the investigation cost, and the reputational damage. Average chargeback cost to merchants: around $110, including fees, investigation, and operational overhead. For issuers, add the card reissue and customer recovery costs and the number climbs past $150.

Flagging legitimate transactions (false positives): A real customer's purchase gets blocked or delayed. They call support. Maybe they abandon the purchase. Maybe they abandon your platform. Aite-Novarica estimates that false declines cost merchants over $443 billion globally per year, dwarfing the roughly $48 billion in actual card fraud losses.

These two failure modes map to two metrics.

Recall (also called sensitivity): of all the actual fraud in your data, what percentage did the model catch? A recall of 0.80 means the model flags 80% of fraud and misses 20%.

Precision: of all the transactions the model flagged as fraud, what percentage were actually fraudulent? A precision of 0.05 means for every real fraud case the model catches, it falsely flags 19 legitimate transactions.

Accuracy blends these into a single number that obscures both. A model with 60% recall and 3% precision can still show 99% accuracy when the fraud rate is low enough. You've caught just over half the fraud while drowning your investigations team in false alerts, and the accuracy metric says everything is fine.

The budget your investigators set

A fraud investigations team is a fixed resource. Say you have 15 analysts, each handling about 40 cases per day. That's 600 investigations per day, roughly 18,000 per month.

Your model needs to select which 18,000 transactions out of 30 million get investigated. If the model has 5% precision, it flags 360,000 transactions per month as suspicious. Your team can only look at 18,000 of those. The other 342,000 sit in a queue or get auto-resolved, which means auto-ignored.

Catching more fraud means flagging more transactions. Your alert queue grows, precision drops, and investigators drown in false alerts. Tightening the queue means letting some fraud through. There's no setting that gives you both.

The right balance depends on the cost ratio. If a missed fraud event costs you $150 and a false positive investigation costs $15 in analyst time, then each false negative is about 10 times more expensive than each false positive. That ratio should directly set your operating point on the precision-recall curve.

Most teams pick their threshold by gut. The ones that get this right pick it by multiplying.

What your threshold actually controls

A fraud model outputs a probability score between 0 and 1 for each transaction. The threshold you choose, the score above which you flag a transaction, determines your position on the precision-recall curve.

Set the threshold at 0.9 and you flag only transactions the model is very confident about. Precision is high, your investigators see mostly real fraud, but recall drops because the model stays quiet on anything below 0.9, including some actual fraud with ambiguous signals.

Set it at 0.3 and you catch nearly everything. Recall climbs toward 95%, but precision craters. Your investigators spend most of their day closing false alerts.

The threshold is a business decision. It encodes how much you're willing to pay in false positives to avoid a false negative. When someone asks "how accurate is the model," they're asking a question that has no answer until someone decides how much a missed fraud case costs relative to a wasted investigation.

Picking the metric that matches the constraint

For fraud detection, the metric that works is precision at a fixed recall target, or recall at a fixed precision target. Pick the constraint that matches your business reality.

If regulators require you to catch at least 80% of fraud (common in banking), fix recall at 0.80 and optimize for precision. The question becomes: given that we must catch 80% of fraud, how clean can we make the alert queue?

If your investigation team can handle 20,000 alerts per month and no more, fix the alert volume and measure what recall you achieve at that budget. The question becomes: given 20,000 investigations, how much fraud can we find?

Both framings force you to state the constraint first and measure the model against it. Neither one is called accuracy.