Evaluating generative models

At 3:03 a.m., a product team checks a new text-to-image checkpoint. The images look polished. They are also, suspiciously, all variations of the same blue mug.

A second checkpoint produces different scenes every time, but it quietly ignores half the prompts. Ask for a red bicycle beside a yellow taxi and you get a beautiful bicycle beside a beautiful bicycle.

Neither model can be judged with ordinary accuracy. There is no single correct image of a mug, and there is no reference photograph that uniquely answers an open-ended prompt. A useful evaluation must answer several questions:

• Quality: do individual samples look plausible?
• Coverage: does the model produce the different kinds of examples found in the data?
• Condition adherence: does the output follow the prompt, class label, or input?
• Risk: does it produce unsafe, biased, copyrighted, or otherwise unacceptable content?

Classic automatic metrics mostly address the first three. They do not make the fourth disappear.

The mental model: compare distributions, not answers

A classifier maps an input to a label. A generator maps random noise, and perhaps a condition, to a sample. Change the noise and a good generator should usually produce another valid sample.

Write the real data distribution as p_data and the generator’s distribution as p_theta. You do not observe either directly. You have a finite real dataset and a finite collection of generated samples, so evaluation estimates how their behavior differs.

A single generated image tells you little. A batch reveals more:

• If every image is ugly, quality is poor.
• If every image is the same attractive image, coverage is poor.
• If images look realistic but ignore the condition, adherence is poor.

A metric such as FID compresses several properties into one number. That helps compare checkpoints, but can hide which property changed. For an unconditional model, quality and coverage are central. For text-to-image, prompt adherence is a separate axis: a realistic image is not necessarily a correct image.

FID: distance between feature distributions

FID, or Fréchet Inception Distance, follows four steps:

1. Pass real and generated images through a pretrained image encoder.
2. Keep an intermediate feature vector for each image.
3. Estimate the mean and covariance of each feature set.
4. Measure the distance between the resulting distributions.

Common implementations use the 2,048-dimensional pool3 features of Inception-v3, although the encoder and preprocessing must be stated. FID is not a raw-pixel distance. A one-pixel translation can produce large pixel error while leaving semantic features similar. Conversely, a change important to a specialist may barely affect ImageNet-trained features.

Let μ_r and Σ_r be the mean and covariance of real features, and μ_g and Σ_g the generated statistics:

FID = ||μ_r − μ_g||² + Tr(Σ_r + Σ_g − 2(Σ_rΣ_g)^(1/2))

Here, μ is the average feature vector, Σ records spread and relationships between feature dimensions, and Tr is the trace. The square root is a matrix square root. This is the squared Fréchet, or squared 2-Wasserstein, distance between two Gaussian distributions in feature space. Lower is better. Zero means the estimated Gaussian feature distributions match; it does not mean every generated image matches a real image.

In one dimension, covariance becomes variance:

(μ_r − μ_g)² + var_r + var_g − 2√(var_r × var_g)

import numpy as np

def fid_1d(mu_r, var_r, mu_g, var_g):
    # Simplified 1D Frechet distance between two Gaussians.
    return (mu_r - mu_g)**2 + var_r + var_g - 2*np.sqrt(var_r * var_g)

real = (0.0, 1.0)

for label, gen in [
    ("bad gen (far)", (3.0, 1.0)),
    ("ok gen", (1.0, 1.0)),
    ("good gen (close)", (0.1, 1.0)),
]:
    print(f"{label:<18} FID={fid_1d(*real, *gen):.2f}")

bad gen (far)      FID=9.00
ok gen             FID=1.00
good gen (close)   FID=0.01

The generated mean moves from 3.0 toward the real mean of 0.0, so the squared mean difference falls from 9.0 to 0.01. Equal variances make the covariance term zero here.

Collapse can also raise FID. If the generator has mean 0 but variance 0, the score is:

0² + 1 + 0 − 2√(1 × 0) = 1

That is worse than the generator with mean 0.1 and variance 1, whose score is 0.01. Repeating one attractive image can therefore look bad in feature statistics.

The Gaussian shortcut is useful, and lossy

FID does not compare the full feature distributions. It fits one Gaussian to each, retaining only first and second moments. That makes the calculation manageable but discards multimodal structure.

Suppose real one-dimensional features have two equally common modes at −2 and +2. Their mean is 0 and variance is 4. A generator with one smooth Gaussian of mean 0 and variance 4 has the same Gaussian summary, so the simplified FID is zero even though the distributions have different shapes.

FID can penalize obvious collapse, but it does not reliably detect every missing mode. Nor is either term a pure quality or diversity measure. FID is an alarm bell, not a diagnosis.

FID is a protocol, not a magic number

A FID score is comparable only when the measurement procedure is comparable. Use a held-out real split; comparing against training data can reward memorization.

Keep these fixed:

• the real split and number of generated samples
• decoding, preprocessing, encoder, and encoder weights
• prompt or class distribution for conditional models
• random seeds, or at least the seed policy

Fifty thousand generated images is a common convention, not a law. A score from 1,000 images is not directly comparable with one from 50,000, and covariance estimates become noisy with too few samples. Repeat important checkpoint evaluations across generation seeds and report variation or bootstrap intervals.

For conditional models, pooled FID can hide failure. A model that generates only plausible landscapes may look good when landscapes and portraits are pooled. Report class or prompt-bucket scores as well as the aggregate.

Inception Score: confident classes, varied batches

Inception Score (IS) uses a classifier’s predictions on generated images:

IS = exp(E_x KL(p(y|x) || p(y)))

It is high when each image receives a confident class prediction and the batch contains varied classes. With a perfect three-class classifier, three images assigned to three different classes have ideal score 3; three images assigned to one class have score 1.

IS never checks the real dataset. Sharp, bizarre images that the classifier labels confidently can score well, and ImageNet labels may be irrelevant to fashion, diagrams, or radiology. FID largely replaced IS for broad comparisons because it compares generated features with real features.

CLIP-score: does the image answer the prompt?

FID cannot tell whether an image follows a text prompt. A realistic cat can improve FID when the prompt requested a dog.

A CLIP-style metric embeds the image and prompt in a shared space. For normalized image vector v and text vector t, the basic score is cosine similarity, often v · t. The model, preprocessing, and any rescaling belong in the experiment log.

CLIP similarity is useful for broad semantic alignment, but can reward generic associations while missing:

• counts, spatial relations, negation, and small text
• exact attributes or colors
• prompt classes hidden by an overall average

Use a fixed prompt suite with buckets for objects, attributes, counts, relations, and text rendering. Report bucket results, then pair them with targeted checks and human preference.

Choosing among the metrics

Choose the metric that exposes the failure you need to see.

KID, or Kernel Inception Distance, is a kernel distance between distributions on Inception features. Its standard estimator is unbiased, which helps at smaller sample counts. It can be slightly negative because an unbiased finite-sample estimate is not forced to be nonnegative.

Feature precision asks how much of the generated set lies near the real feature manifold; recall asks how much of the real manifold the generated set reaches. These are not ordinary classification precision and recall, and implementations differ in how they define neighborhoods.

A strong image-generation report usually combines a fixed held-out set, FID or KID where the feature space is defensible, a coverage measure, conditional adherence tests, and blind human comparisons. Model-based judges can triage thousands of examples, but calibrate them against human judgments before using them to gate a release.

Human evaluation that is worth the money

Human evaluation is expensive, but a careless metric can send weeks of training in the wrong direction.

Pairwise comparison is often easier than absolute scores. Show the same prompt with outputs from models A and B, randomize left and right, hide model names, and ask one focused question such as, “Which image better follows the prompt?”

Separate dimensions when needed:

• visual quality
• prompt adherence
• artifact severity
• usefulness for the task

Report preference rates with uncertainty over prompts. If A wins 62%, also report the number and diversity of prompts, the rater protocol, and an interval showing stability. Bootstrap over prompts rather than treating many votes on one prompt as independent examples.

At $0.05 per judgment, 10,000 judgments cost $500 before overhead and quality control. Expert review costs more, but may be necessary for medical, legal, safety, or brand-sensitive outputs. Human scores are not perfect; they are simply closer to the product question when that question is perceptual preference.

A production evaluation pattern

Treat evaluation like a test suite, not a screenshot gallery.

Freeze representative prompts and real examples, including rare classes, long prompts, negation, multiple objects, unusual compositions, and safety cases. Log model versions, samplers, guidance settings, and seeds. Run automatic metrics on the full set, inspect slices, and send a randomized subset to blind reviewers.

Keep a private holdout for final decisions if the public set has become a training target. Store the generated samples too: a score without its artifacts is hard to debug.

When not to use FID alone

Do not use FID as the sole gate for a custom domain. ImageNet features may poorly represent pathology, diagrams, faces, or typography. A domain-specific encoder can help, but validate it; otherwise a small expert review may be more informative than a precise-looking decimal.

FID also does not answer safety, memorization, latency, or cost questions. A model can have excellent FID while copying training images, generating unsafe content, taking twelve seconds per sample, or costing too much to serve. Test those properties separately.

For GANs and diffusion models, the same evaluation logic applies. For video, 3D, and audio generation, framewise image FID is incomplete because it can reward plausible frames that flicker, drift, or violate motion and sound structure.

In one breath

• Open-ended generators have many valid outputs, so evaluation compares real and generated distributions rather than per-sample answers.
• FID compares Gaussian summaries of Inception features. Lower is better, but the result depends on the encoder, preprocessing, sample count, and finite-sample noise.
• FID can penalize obvious collapse yet miss missing modes and cannot cleanly separate quality from coverage.
• IS rewards confident, varied classifier predictions without checking real data. CLIP similarity adds prompt adherence but can miss exact relations, counts, and attributes.
• KID, precision and recall, targeted prompt tests, and blind human preference fill different gaps. A fixed protocol beats a fashionable single number.

Quick check

Next

These metrics score the generative models behind GANs and diffusion. For GANs, watch FID alongside mode collapse. For diffusion, flow matching, and other samplers, keep the evaluator fixed while the generator changes. That is how a score becomes evidence instead of decoration.