Evaluating generative models

A classifier has a right answer, so you score it with accuracy. A generative model doesn’t — there’s no single correct image of “a cat.” So how do you measure it? You need two things at once: quality (do the samples look real?) and diversity (do they cover the whole data distribution, or has the model collapsed to a few outputs?). The metrics work at the level of distributions, not individual samples.

FID: distance between feature distributions

The standard metric is FID (Fréchet Inception Distance). Pass a big batch of real images and a batch of generated images through a pretrained Inception network to get feature vectors. Model each set’s features as a Gaussian and measure the Fréchet distance between the two Gaussians. Lower is better — the generated feature distribution is closer to the real one:

import numpy as np

def fid_1d(mu_r, var_r, mu_g, var_g):       # simplified 1D Frechet distance between two Gaussians
    return (mu_r - mu_g)**2 + var_r + var_g - 2*np.sqrt(var_r * var_g)

real = (0.0, 1.0)                            # real features: mean 0, variance 1
for label, gen in [("bad gen (far)", (3.0, 1.0)), ("ok gen", (1.0, 1.0)), ("good gen (close)", (0.1, 1.0))]:
    print(f"{label:<18} FID={fid_1d(*real, *gen):.2f}")

bad gen (far)      FID=9.00
ok gen             FID=1.00
good gen (close)   FID=0.01

As the generated distribution moves toward the real one, FID falls from 9.0 to ~0. Crucially FID captures both failure modes: poor quality shifts the feature mean (the (μ_r − μ_g)² term), and mode collapse shrinks the feature variance (the covariance terms), so a model producing one beautiful image repeatedly still scores badly. That’s why FID became the default for image generation.

Inception Score, and CLIP-score for prompts

Two more metrics fill gaps FID doesn’t:

• Inception Score (IS) — the older metric, from a classifier’s predictions on generated images: high when each image is confidently one class (quality) and the classes are varied across samples (diversity). Its flaw: it never looks at the real data, so it can’t catch a model that’s realistic-but-wrong-distribution, and it’s gameable. FID, which compares to real data, largely superseded it.
• CLIP-score — for text-to-image, FID says nothing about whether the image matches the prompt. CLIP-score measures exactly that: the CLIP similarity between the generated image and the prompt text. A model can have great FID (realistic images) and poor CLIP-score (ignores the prompt) — you need both: FID for realism, CLIP-score for adherence.

In one breath

• Generative models have no single right answer, so you can’t use accuracy — you measure quality and diversity at the distribution level.
• FID passes real and generated images through Inception, models each feature set as a Gaussian, and takes the Fréchet distance — lower is better (the demo: 9.0 → 0.01 as gen approaches real); it catches both poor quality (mean shift) and mode collapse (variance shrink).
• Inception Score rates confident-yet-varied class predictions but never sees real data (so it’s weaker and gameable); FID largely replaced it.
• CLIP-score measures prompt adherence for text-to-image (CLIP similarity of image to prompt) — you need FID for realism + CLIP-score for adherence.
• All are proxies on biased Inception features; human preference (side-by-side, Elo) is the gold standard — automatic metrics to iterate, humans to validate (the BLEU/ROUGE lesson again).

Quick check

Next

These metrics score the generative models you’ve met — GANs (watch FID for mode collapse) and diffusion. CLIP-score builds on CLIP embeddings, and like BLEU/ROUGE these stay proxies for human judgment.