Anomaly detection

At 3:07 a.m., a payment platform sees 40,000 card authorisations. One is for $1,200 from a card that usually buys $8 coffees. Another is for $42, which looks ordinary until you notice it happened two minutes after the previous purchase in a different country. A third is perfectly legitimate: a customer has just arrived for a week-long holiday.

There are no reliable labels saying which transaction is fraud. Chargebacks arrive weeks later, and most unusual events are merely unusual. You still need a shortlist for an analyst to inspect now.

Anomaly detection assigns each observation an anomaly score, expressing how poorly it fits a learned pattern, and ranks observations by that score. It is usually unsupervised: training does not receive a “fraud” or “not fraud” column.

An anomaly detector does not discover truth. It discovers mismatch. A flagged transaction may be fraud, a broken sensor, or a valuable customer travelling for the first time.

The mental model: unusual means unusual relative to something

Start with the data, not the algorithm.

Suppose each card transaction has two features:

• transaction amount;
• minutes since the card’s previous transaction.

Most everyday purchases sit around $8 to $60, with gaps from 30 minutes to a day. A legitimate travel pattern forms a second group around $300 to $900, with gaps of one to four hours.

A $1,200 purchase with a two-minute gap is globally unusual: it is far from both groups. A global anomaly detector should give it a high score.

A $42 purchase with a two-minute gap is different. Its amount is ordinary, but its combination with the tiny time gap is strange among nearby everyday purchases. It is a local anomaly: not far from the whole dataset, but unlike its neighbours.

The useful question is:

Which observations are easy to separate from the observations that resemble them?

This produces a ranking, not a verdict. A team might investigate the top 20 transactions per hour or everything above a threshold that yields no more than 200 alerts per day. The threshold belongs to the operation; the model supplies the ordering.

Isolation Forest: isolate the lonely point

Isolation Forest builds many random decision trees. Each split chooses a feature and threshold at random. A point’s path length is the number of splits needed to reach a leaf.

Imagine 100 normal transactions packed into amounts from $8 to $60. A $1,200 transaction can be separated by an early split such as amount <= 630. The remaining normal points need more splits to separate from one another, so the expensive transaction is isolated quickly.

The mechanism is:

1. A random tree picks a feature and cut.
2. A point on its own side reaches a leaf sooner.
3. Many trees average the path lengths.
4. A short average path becomes a high anomaly score.

The algorithm does not calculate distance to a centroid, need clusters, or require labelled fraud. Its assumption is that anomalies are few and different enough to be separated by random cuts. Random subsets of training rows keep trees cheap and less correlated.

In scikit-learn, IsolationForest.predict returns -1 for an outlier and 1 for an inlier. decision_function is positive for inliers and negative for outliers; lower score_samples means more abnormal. None of these scores is a fraud probability.

Isolation Forest is a fast first baseline for numeric tabular data and is especially useful for global anomalies such as the $1,200 transaction. Its limitation is the rarity assumption: if 5,000 fraudulent transactions form a dense, repeated pattern, each may look normal because the attack has become common.

Local Outlier Factor: compare neighbourhoods, not the whole map

Local Outlier Factor, or LOF, asks:

Is this point much less dense than its own neighbours?

LOF compares a point’s local density with that of its k nearest neighbours. A simplified numerical example makes the ratio concrete. Suppose a transaction has five neighbours. After LOF’s distance correction, its average neighbour distance is 4 units, so its local reachability density is 1 / 4 = 0.25. Suppose those neighbours each sit in a region with density 1. Its LOF is approximately:

average neighbour density / point density = 1 / 0.25 = 4

A LOF near 1 means similar density to the neighbours; a value well above 1 means the point is substantially sparser. The implementation uses reachability distance, which prevents one accidentally close neighbour from dominating:

local density = 1 / average reachability distance

LOF = average neighbour density / point density

The $42 transaction is ordinary in amount, but a two-minute gap can place it in a sparse pocket beside a dense everyday cluster. LOF can raise it even when a global method considers the amount harmless.

The choice of k determines what “local” means. A small value notices tiny pockets but is noisy; a large value is steadier but can wash out a small anomaly or mix the everyday and travel groups.

LOF is distance-based, so feature units matter. If amount ranges from 0 to 10,000 and time from 0 to 1,440 minutes, raw Euclidean distance will be dominated by amount. Standardise or robustly scale numeric features, then check that the geometry represents a meaningful notion of “near”. In high-dimensional spaces, nearest neighbours can become unreliable: this is the curse of dimensionality.

LOF is generally more expensive than Isolation Forest at large scale. In scikit-learn’s default outlier mode, fit_predict returns -1 for flagged rows and 1 for normal rows; negative_outlier_factor_ becomes more negative as a point looks more abnormal. This is a ranking signal, not a calibrated probability.

A small working example

This toy dataset has 200 points near the origin and 15 points spread across a larger square. It mirrors the card example without pretending that a square is a realistic fraud generator.

contamination=0.07 tells each estimator to use a cutoff corresponding to about 7 percent of the 215 rows. The count will therefore be around 15, but it is not a recall measurement: the detectors do not know which rows came from the second distribution.

import numpy as np

from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor

rng = np.random.default_rng(0)
normal = rng.normal(0, 1, size=(200, 2))
outliers = rng.uniform(-6, 6, size=(15, 2))
X = np.vstack([normal, outliers])

# contamination sets the fraction used to choose the flagging cutoff
iso = IsolationForest(contamination=0.07, random_state=0).fit(X)
pred_iso = iso.predict(X)            # -1 = anomaly, 1 = normal
print(f"Isolation Forest flagged {(pred_iso == -1).sum()} anomalies")

# Default LOF mode finds outliers in the data passed to fit_predict
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.07)
pred_lof = lof.fit_predict(X)
print(f"LOF flagged             {(pred_lof == -1).sum()} anomalies")
print(f"\n(15 true outliers were injected.)")

Outlier detection versus novelty detection

These names describe the data available during fitting.

Outlier detection fits on potentially contaminated data: normal observations and unknown outliers may be mixed. It can rank rows in that history and, for Isolation Forest, score future rows with predict or decision_function. scikit-learn’s default LOF mode instead identifies outliers among rows passed to fit_predict; it cannot score unseen rows unless constructed with novelty=True.

Novelty detection fits on a clean sample of normal data, then scores future observations against that baseline. A new $1,200 transaction can be novel even if it was absent during training.

The distinction matters when an attack has contaminated the training history. An outlier detector may learn that attack as normal. A clean novelty baseline, when available, avoids that problem. With scikit-learn LOF, use LocalOutlierFactor(novelty=True) with clean normal data, then score new observations; do not treat that as equivalent to default fit_predict.

One-Class SVM is another novelty option for moderate-sized, well-scaled data. A simple z-score, IQR rule, or domain limit is easier to audit for one well-behaved measurement. Once trustworthy labels exist, a supervised classifier can optimise the actual target—fraud, failure, or chargeback—instead of generic unusualness.

The production pattern

First define the event and decision horizon. Score a payment using only information available at authorisation. A chargeback field, future customer action, or feature aggregated over the next 24 hours is data leakage.

Build behavioural features rather than relying on amount alone: amount relative to the card’s usual amount, time since the previous transaction, country transition, device change, and recent transaction count. Fit transformations such as scaling on the training period only, as with feature engineering.

Fit on a historical window and validate on a later one. Random splits can hide seasonality and drift. For every scored row, retain the model score, feature snapshot, model version, and reason codes or nearest neighbours so an analyst can answer “why this one?”

Finally, make ranking an operating policy. You might review the top 0.2 percent, block only the top 0.01 percent, and send the middle to step-up authentication. Those thresholds have different costs, so choose them using missed incidents, review capacity, customer friction, and reviewed outcomes. A single contamination value is rarely the whole policy.

Monitor score distributions and alert rates by customer segment, country, merchant, and time period. A holiday sale or device rollout can shift the baseline; a sudden alert-rate increase may be a data-pipeline problem before it is a crime wave.

Failure modes you will actually see

The alert queue floods after a deploy. A jump from 200 alerts per day to 18,000, concentrated in one country or app version, often means a changed unit, a missing feature replaced with zero, or a new seasonal regime. Compare feature distributions, units, and missingness before raising the threshold. Segment or retrain on a representative recent window.

Known incidents are not flagged. A coordinated attack may be a dense group, not a lonely point, or it may already be present in the training history. Use a clean novelty baseline where possible, add sequence or group-level features, and move to supervised learning when incident labels become dependable.

LOF changes its mind after preprocessing changes. That is expected when a new high-range feature changes the distance geometry. Scale numeric features, inspect nearest neighbours, and test reasonable n_neighbors values. If “near” has no business meaning, LOF is the wrong tool.

The detector finds strange things but not bad things. Anomaly scores answer “how different is this from the reference pattern?”, not “should we block it?” Keep those decisions separate: use scores for triage, add business context, and measure precision among reviewed cases. With labels, choose metrics and thresholds based on real costs; see the metrics lesson.

Quick check

Next

Anomaly detection sits beside clustering in the unsupervised toolkit. When reliable labels arrive, compare it with supervised alternatives before shipping a detector.