How do you approach anomaly detection, and why is accuracy a bad metric for it?

How to think about it

The crisp answer

I treat anomaly detection as a ranking and thresholding problem: learn normal behavior or a boundary around it, assign each new observation an anomaly score, and choose the alert threshold from the business cost of misses and false alarms. Accuracy is misleading because anomalies are usually rare, so a model that predicts “normal” for everything can score almost perfectly while detecting nothing.

Why accuracy fails

Suppose we monitor one million card transactions and historical investigation shows that 1,000 are fraudulent. Fraud is therefore 0.1 percent of the data.

An “always normal” system gets 999,000 legitimate transactions right. Its accuracy is 99.9 percent. It also catches zero fraud.

That is not a useful detector. It is a calculator that has learned to avoid raising its hand.

Here is the same dataset with two possible thresholds. A true positive is fraud correctly flagged. A false positive is a legitimate transaction incorrectly flagged. A false negative is fraud that the system misses.

Decision rule	Flagged	True positives	False alarms	Missed fraud	Precision	Recall	Accuracy
Always normal	0	0	0	1,000	Not defined	0%	99.90%
Threshold A	500	300	200	700	60%	30%	99.91%
Threshold B	2,000	600	1,400	400	30%	60%	99.82%

Precision means “of the alerts, how many were truly anomalous?” It is calculated as true positives / all positive predictions. Recall means “of all the real anomalies, how many did we catch?” It is true positives / all actual anomalies.

Threshold A is slightly more accurate than the useless always-normal system, even though it misses 700 fraudulent transactions. Threshold B has lower accuracy but catches twice as much fraud as Threshold A. Accuracy is answering the wrong question because the 999,000 normal cases dominate the arithmetic.

I would usually report precision, recall, and F1 for a chosen operating point. F1 is the harmonic mean of precision and recall, so it penalizes a system that is excellent at one and terrible at the other. I would also inspect the precision-recall curve and its area under the curve, or PR-AUC, when the positive class is very rare.

ROC-AUC is not forbidden. It measures how well the model ranks anomalies above normal examples across thresholds. But under extreme imbalance, it can look reassuring while the alert queue is unusable. A false-positive rate of only 0.1 percent creates about 999 false alarms among 999,000 normal transactions. That can be roughly as many as the actual fraud cases.

What the detector is actually doing

An anomaly is not simply a rare observation. It is an observation that is unusual relative to a relevant reference population and context. A $10,000 payment may be ordinary for a corporate account and highly unusual for a student account. A spike in web traffic may be an attack on a weekday but normal during a product launch.

Most anomaly detectors produce an anomaly score, a number that ranks observations from more normal to more suspicious. The score is not automatically a probability of fraud. I then choose a threshold, the cutoff above which an observation becomes an alert.

That distinction matters. The model may rank 10,000 transactions correctly, but the business may only have investigators for 500 alerts. The useful question is not merely “Which model has the best score?” It is “Which threshold gives us an acceptable number of useful alerts?”

The method depends on the available labels:

• Statistical methods describe normal behavior with quantities such as a median, interquartile range, quantiles, or a probability distribution. A value far outside the usual range becomes suspicious. These are fast and explainable, but a Gaussian assumption can be badly wrong for skewed transaction amounts.
• Distance and density methods compare an observation with nearby observations. A point far from its neighbors, or sitting in a sparse local region, can be anomalous. Feature scaling matters because a dollar-valued feature can otherwise overwhelm a small but important frequency feature.
• Isolation Forest repeatedly partitions the feature space with random splits. Rare, isolated points tend to be separated in fewer splits, so they receive a more anomalous score. It is a useful baseline for tabular data, but it does not understand causality or business context by itself.
• One-class SVM learns a boundary around examples considered normal. It can work well in moderate-dimensional, carefully scaled data, but its result is sensitive to kernel and boundary settings.
• Autoencoders learn to reconstruct normal examples and flag high reconstruction error. They are useful for complex signals, but a sufficiently expressive autoencoder may reconstruct anomalies too, which weakens the signal.
• Time-series detectors model trend, seasonality, and recent history, then flag unusually large residuals. A simple global outlier detector often calls every Monday morning spike anomalous, which is not a model; it is a complaint about calendars.

There are also different kinds of anomalies. A point anomaly is one unusual event, such as a login from an impossible location. A contextual anomaly is unusual only in context, such as a normal volume of traffic arriving at 3 a.m. A collective anomaly is a suspicious sequence whose individual points look ordinary, such as a slow credential-stuffing campaign.

How I would approach it in production

1. Define the decision before choosing the algorithm

I would first clarify what one observation represents, when a decision must be made, and what “anomaly” means operationally.

For fraud, the observation might be one transaction. For infrastructure, it might be a five-minute service window. For customer behavior, it might be one customer-day. Mixing these units can create misleading results.

I would also define the label and its delay. A chargeback might confirm fraud weeks after the transaction, so today’s recall cannot be measured using information that does not exist yet.

2. Establish a simple baseline

I would begin with rules and robust statistics: transaction amount above a known percentile, an unusual login country, or a residual outside a historical band. A baseline gives us a comparison and often exposes data problems faster than a sophisticated model.

If an Isolation Forest improves an elaborate offline metric but produces alerts that investigators cannot explain, the baseline may still be the better product.

3. Build features from information available at decision time

This is where leakage commonly enters. A feature such as “number of chargebacks in the next 30 days” makes a model look brilliant during training and is unavailable when the payment arrives.

For the payment example, useful features might include amount relative to the customer’s recent median, transaction velocity, distance from the previous login, device novelty, and merchant history. I would preserve the context rather than relying only on global rarity.

4. Validate chronologically

A random train-test split can place nearly identical transactions from the same customer on both sides of the split. It can also let future behavior influence the apparent past.

I would train on earlier data, tune the threshold on a later validation period, and test on the most recent period. If labels arrive after 45 days, I would respect that delay when constructing the split.

Then I would choose the threshold using the actual operating constraints. Suppose a false alarm costs $3 in analyst time and a missed fraud costs $200. Threshold A costs approximately 200 × 700 + 3 × 200, or $140,600, under those assumptions. Threshold B costs 200 × 400 + 3 × 1,400, or $84,200. B is preferable here, despite lower precision, because missing fraud is much more expensive.

If a false positive automatically declines a good customer’s payment, its cost may be far higher than $3. The preferred threshold can change completely. F1 cannot express that business trade-off because it assigns no dollar value to either error.

5. Monitor the decision, not just the model

In production I would track alert rate, score distribution, precision among reviewed alerts, delayed recall when labels arrive, investigation backlog, and detection latency.

I would also monitor feature drift. A new payment flow, a seasonal event, or a currency change can shift normal behavior and suddenly turn yesterday’s threshold into a factory for false alarms. Retraining should be tied to evidence about changing behavior, not an automatic calendar ritual.

The senior-level nuance

Anomaly detection is often described as unsupervised because labels are scarce. That describes the training signal, not the evaluation situation. An unsupervised model can still be evaluated with later fraud labels, sampled human reviews, and carefully designed backtests.

If we have thousands of reliable labels for a known fraud pattern, I would compare the anomaly detector with a supervised imbalanced classifier. A supervised model may identify known fraud more accurately. An anomaly detector can still be valuable as a second signal for novel attacks, data quality failures, or behavior that was absent from the training labels. In practice, a hybrid system is often stronger than an ideological choice between “supervised” and “unsupervised.”

Accuracy is not mathematically useless. It can be reasonable when classes are balanced, error costs are similar, and the prediction task genuinely values both classes equally. Those conditions are unusual in anomaly detection. The metric is bad here because the prevalence is low and the consequences are asymmetric.

Failure modes I would look for

• Contaminated normal data: the training set called “normal” contains many anomalies. The model absorbs unusual behavior into its reference pattern. The first symptom is often a suspiciously low alert rate and poor ranking of known incidents.
• A threshold tuned for a spreadsheet rather than operations: offline precision looks fine, but the deployed system sends 20,000 alerts per day to a team that can review 500. The alert backlog grows before anyone notices the metric problem.
• Random validation leakage: test performance is excellent, then production performance collapses after deployment. Repeated customers, future features, or temporal overlap may have made the test set unrealistically easy.
• Feedback loops: only flagged transactions are investigated, so labels become biased toward the current model’s beliefs. The system can become increasingly confident about a narrow slice of reality.
• Normal behavior drift: a product launch or seasonal event causes a flood of alerts. Automatically treating every alert as new training data can teach the model that a real attack is normal.

What they’ll ask next

Would you use supervised or unsupervised learning?
It depends on labels. With few or delayed labels, I would start with a normality model or an unsupervised ranker. With enough trustworthy labels, I would test a supervised imbalanced classifier and possibly combine it with an anomaly score for unknown patterns.

Which metric would you choose?
I would report precision and recall at the operating threshold, PR-AUC for ranking quality, and alert volume or recall at the team’s fixed capacity. I would choose the threshold from false-positive and false-negative costs, not from the threshold that happens to maximize F1.

How do you evaluate a detector when there are no labels?
I would not pretend to know recall. I would use delayed outcomes where available, random samples of unflagged cases, expert review, stability checks, and monitoring of alert quality over time. Synthetic anomalies can test whether the pipeline responds to known perturbations, but they do not prove that real anomalies are being found.

Say this in the interview

“I model or learn normal behavior, rank new cases by anomaly score, and set the threshold using operational cost and capacity; accuracy fails because an all-normal model can be nearly perfect on paper while catching zero of the rare events we care about.”