How do you handle severe class imbalance when training a deep learning model?

How to think about it

If one in 1,000 card transactions is fraud, a model that predicts the negative class — legitimate — for everything scores 99.9% accuracy and catches nothing. I handle severe class imbalance, meaning one class is much rarer than the other, by keeping evaluation realistic, making minority mistakes matter during training, and choosing a decision threshold from the business cost rather than from accuracy.

Why imbalance breaks the obvious approach

Suppose a training set contains 999 legitimate transactions and one fraudulent transaction for every 1,000 rows. In binary classification, fraud is the positive class, and legitimate transactions are the negative class.

An all-negative model has this confusion matrix over 100,000 transactions containing 100 fraud cases:

	Predicted fraud	Predicted legitimate
Actual fraud	0	100
Actual legitimate	0	99,900

Its accuracy is 99.9%. Its fraud recall — the fraction of actual fraud it catches — is zero. Its F1 score is also zero.

The problem appears in the training objective as well. With ordinary binary cross-entropy, the loss is averaged across examples. A model can reduce that average loss substantially by becoming very good at the 999 easy negatives while remaining useless on the one positive. The positive example is not invisible, but its influence is diluted.

Mini-batches make this worse. With a 0.1% positive rate and a batch size of 256, a randomly drawn batch contains no positive example about 77% of the time. Many updates therefore tell the model only, “keep predicting negative.”

That is why I separate two questions:

1. What should the model learn from? This is where loss weighting, sampling, and focal loss help.
2. What prediction should trigger action? This is where threshold selection, calibration, and business costs matter.

Changing the training distribution does not automatically answer the second question.

The production pattern I would use

1. Split the data without leaking the future

I would first define how a prediction will be made in production and reproduce that information boundary in the split.

For fraud, a random row-level split is often unsafe. The same card, account, device, merchant, or IP address may appear in both training and validation. A model can memorize those identities and produce an impressive validation score that disappears on new customers. Fraud labels may also arrive days later, so a feature created after the transaction is a form of target leakage.

I would usually prefer a time-based split, with older transactions for training and newer transactions for validation and testing. If the data is genuinely independent and identically distributed, stratification helps ensure that each split contains enough positives. Either way, validation and test data should retain the real fraud rate. Do not make the test set 50% fraud merely because it is easier to inspect.

For the same reason, I would never duplicate minority rows into validation or test data. Oversampling belongs in the training pipeline only.

2. Start with weighted loss

For binary classification, I would establish a class-weighted binary cross-entropy baseline. In PyTorch, BCEWithLogitsLoss combines the sigmoid operation and binary cross-entropy in a numerically stable implementation.

For a training split with 799,200 legitimate transactions and 800 fraud cases, the initial positive weight is 999:

import torch
import torch.nn as nn

n_neg, n_pos = 799_200, 800

pos_weight = torch.tensor(
    [n_neg / n_pos],
    dtype=torch.float32,
    device=device,
)

criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)

logits = model(features).squeeze(-1)
loss = criterion(logits, labels.float())

pos_weight multiplies the loss contribution from positive examples. It does not create new fraud data, and it does not tell the model that every transaction has a 50% chance of being fraudulent.

The negative-to-positive ratio is a reasonable starting point, not a law. A weight of 999 may produce excellent recall and an unusably large number of false alerts. I would tune it against the metric that reflects the deployment goal. For a multiclass problem, the equivalent baseline is a class-weight vector passed to CrossEntropyLoss.

3. Use sampling when batches contain too few positives

Loss weighting makes a positive example influential when it appears. Sampling makes it appear often enough to provide a useful gradient.

A weighted sampler can draw minority examples more frequently:

from torch.utils.data import DataLoader, WeightedRandomSampler

labels = torch.as_tensor(y_train, dtype=torch.long)
class_counts = torch.bincount(labels)

sample_weights = (1.0 / class_counts.float())[labels]

sampler = WeightedRandomSampler(
    weights=sample_weights.double(),
    num_samples=len(labels),
    replacement=True,
)

train_loader = DataLoader(
    train_dataset,
    batch_size=256,
    sampler=sampler,
)

This produces roughly balanced class draws over an epoch. It does not guarantee that every individual batch has exactly the same number of positives.

Oversampling is useful when the positive class is so rare that ordinary batches frequently contain none. Undersampling can be useful when the negative class contains millions of repetitive, uninformative examples. I would be careful, though: random undersampling can discard legitimate negatives near the decision boundary. Hard-negative mining — retaining negatives the model frequently confuses with fraud — is often more useful than throwing negatives away uniformly.

I would not blindly combine a balanced sampler with a positive loss weight of 999. The sampler has already increased the relative frequency of positive examples by roughly 999 times. Adding the full class weight can overcorrect and produce a flood of false positives. I would begin with one intervention, measure it, and combine them only deliberately.

4. Try focal loss when easy examples dominate

Focal loss changes the emphasis within a batch. It down-weights examples the model already classifies confidently, so the loss spends more attention on difficult examples.

The usual form is FL = -alpha_t * (1 - p_t)^gamma * log(p_t), where p_t is the probability assigned to the true class, gamma controls how strongly easy examples are down-weighted, and alpha_t can provide class balancing.

With gamma = 2, an example with p_t = 0.99 receives a modulating factor of 0.0001, while an example with p_t = 0.5 receives a factor of 0.25. Easy examples have not vanished, but they contribute much less.

Focal loss is a sensible candidate when the majority class contains huge numbers of easy negatives, as in object detection or some fraud systems. It is not magic. If the minority examples are also easy, focal loss down-weights them too unless the alpha term is chosen appropriately. It can also make probability calibration worse and introduce another pair of hyperparameters. I would compare it with weighted cross-entropy rather than assuming it must win.

A concrete threshold decision

Suppose the test set contains 100,000 transactions, including 100 fraud cases. The following numbers are illustrative, but they show why a threshold is a business decision.

Threshold	Precision	Recall	F1	Alerts
0.50	20%	60%	0.30	300
0.90	66.7%	40%	0.50	60

At the lower threshold, the system catches 60 fraud cases but sends 300 alerts for review. At the higher threshold, it catches only 40 but sends 60 alerts, with two-thirds of those alerts being genuine fraud.

Neither threshold is universally correct. If the review team can investigate only 60 cases, the second operating point may be appropriate. If missing a fraudulent payment costs far more than reviewing a legitimate one, the first may be preferable. In some systems I would choose a threshold that achieves a minimum precision or recall; in others I would rank transactions and review the top fixed number each hour.

I would report precision, recall, F1, and area under the precision-recall curve, or PR AUC. PR AUC measures ranking quality across thresholds and is usually more informative than ROC AUC when positives are rare. The random-ranking baseline for PR AUC is approximately the positive prevalence, so a 0.1% problem needs a very different interpretation from a balanced benchmark.

MCC, the Matthews correlation coefficient, is another useful summary because it uses all four confusion-matrix cells and remains informative under skew. I would still include the confusion matrix at the chosen operating point. A single scalar metric cannot tell an operations team whether it will receive 50 alerts or 50,000.

The senior-level nuance: scores are not automatically probabilities

Balanced sampling and weighted loss change what the model sees during optimization. The resulting score may rank examples well without being calibrated to the real-world prior.

A model output of 0.80 does not necessarily mean “an 80% chance of fraud” after training on artificially balanced batches. If a downstream system uses probabilities for pricing, reserves, or automated declines, I would calibrate on an untouched validation set with the natural prevalence. Platt scaling or isotonic regression are common choices, but calibration itself needs enough positive examples and must respect time and entity boundaries.

I would also inspect performance by merchant, geography, device type, customer tenure, and time period. A model can have acceptable overall recall while missing an entire segment. Severe imbalance often hides segment imbalance inside it.

Common failure modes

The first symptom is perfect-looking accuracy and zero recall. The model has learned the majority-class shortcut. Check the confusion matrix and the number of positive labels per batch before changing the architecture.

The alert queue suddenly becomes ten times larger after adding class weights. The model may have gained recall at the cost of precision, or the threshold of 0.5 may no longer be appropriate. Re-select the threshold on natural-distribution validation data.

Training recall reaches 100%, but held-out minority recall collapses. Repeated minority rows have been memorized. Reduce duplication, use augmentation only when it preserves the label semantics, add regularization, and split by entity so near-duplicates cannot cross the boundary.

Validation PR AUC is excellent, but production performance collapses. Suspect leakage, a time shift, label delay, or a changed positive rate before blaming focal loss. In fraud, “future” features and repeated account identities are particularly common culprits.

What they’ll ask next

“Would you use SMOTE?”

Not automatically. SMOTE creates synthetic points between minority examples and can be reasonable for some tabular data. It is usually inappropriate for raw text, images, or sequential transactions unless the generated examples are demonstrably valid. A synthetic transaction combining incompatible merchant, country, and device features may teach the model a pattern that never occurs. I would establish weighted loss and carefully controlled sampling first.

“Why not use ROC AUC?”

ROC AUC can remain high because the model correctly ranks a huge number of negative examples, even while precision is poor at the operating region that matters. PR AUC focuses attention on the positive class and the false positives that consume the alert budget. I would use ROC AUC as a secondary diagnostic, not the decision metric.

“How do you choose the class weight and threshold?”

I start the positive weight at the negative-to-positive count ratio, then tune it using a validation objective tied to deployment. I choose the threshold separately from the weight, using expected costs, review capacity, or a target precision or recall. The test set remains untouched until the final comparison.

Say this in the interview

“I keep validation and test data at the real prevalence, use weighted loss or carefully controlled sampling so minority errors affect training, compare focal loss when easy negatives dominate, and select the final threshold from precision-recall and business cost rather than trusting accuracy.”