t-SNE & UMAP

At 3 a.m., a support engineer opens a dashboard containing 12,000 customer tickets. Each ticket has been turned into a 768-number embedding, a vector whose numbers encode its meaning. The engineer wants to know whether the data contains separate themes: billing, login problems, cancellations, and perhaps a category nobody has named yet.

A table with 768 columns is not going to help much. A PCA plot gives two axes, but the billing and cancellation tickets overlap in a greyish blob. That does not prove the themes are absent. PCA preserves directions with the most overall variance, not the small curved neighborhoods that may distinguish one theme from another.

t-SNE and UMAP are nonlinear dimensionality-reduction methods: they turn high-dimensional vectors into a small number of coordinates, usually two, while trying to keep nearby points near each other. Their plots can expose local structure that PCA hides.

They can also manufacture a convincing story from weak evidence. The important skill is knowing what the map is allowed to tell you.

The mental model: preserve neighborhoods

Imagine each ticket as a house in a city that has 768 dimensions instead of two. Distance measures dissimilarity according to a chosen metric: a smaller distance means the points are more similar or closer. “Nearby” tickets might use similar words, describe the same product, or share semantic meaning despite different wording.

The methods ask:

For each point, which other points are its neighbors, and how strongly?

They then arrange the points on a 2D sheet so that as many important neighbor relationships as possible survive. The result is a map of local company, not a faithful satellite photograph of the entire city.

A visible island is not automatically a real cluster. It is a claim to check against the original vectors, domain knowledge, and other settings.

A worked example with real arithmetic

Return to the support tickets. Suppose the embedding model puts three tickets at these distances:

• Ticket A: “I was charged twice”
• Ticket B: “The same purchase appears two times”
• Ticket C: “I cannot reset my password”

A and B should be neighbors. C should be farther away. For a toy calculation, pretend the distance from A to B is 1, the distance from A to C is 4, and a Gaussian neighborhood uses a bandwidth of 2.

The unnormalised affinity from A is based on:

weight = exp(-distance² / (2 × bandwidth²))

So A to B gets approximately exp(-1 / 8) = 0.8825, while A to C gets approximately exp(-16 / 8) = 0.1353. After normalising those two weights, A assigns about 0.867 of its neighborhood probability to B and 0.133 to C.

The exact values are not the point: a nearby point receives much more probability than a distant point. The algorithm now has something concrete to preserve while searching for 2D positions.

For the real ticket dataset, cosine distance is often appropriate for text embeddings, especially after normalising each vector to length one. It focuses on the angle between vectors: whether tickets point in a similar semantic direction rather than whether one embedding has a larger magnitude. For ordinary numeric measurements, Euclidean distance may be reasonable after scaling the columns. A map cannot repair a distance measure that was wrong before the map began.

The methods do not need ticket labels. Colouring the final plot by “billing” or “login” can help inspect it, but tuning settings until a known label separates beautifully turns exploration into confirmation bias.

What t-SNE actually optimises

t-SNE, short for t-distributed stochastic neighbor embedding, converts neighborhoods into probabilities.

For every point, it chooses a Gaussian width that gives the neighborhood a chosen perplexity. Perplexity is an effective neighborhood size, not a hard cutoff. It can be written as 2^H, where H is the neighborhood’s entropy in bits. A distribution spread across many neighbors has higher perplexity than one concentrated on a few. t-SNE then symmetrises the point-to-point probabilities so each pair relationship can be optimised in both directions.

Next it places every point in 2D. In that map, it uses a heavy-tailed Student-t distribution rather than a Gaussian to measure how compatible two positions are. The heavy tail gives distant pairs more probability than a Gaussian would, so non-neighbors can be pushed farther apart while genuine neighbors remain close. This helps relieve the crowding problem: many high-dimensional neighbors cannot all fit comfortably around one 2D point.

The objective is the Kullback–Leibler divergence, written as KL(P || Q). P contains the high-dimensional neighborhood probabilities and Q contains those implied by the current 2D layout. Minimising it strongly punishes the map when a high-probability neighbor in P receives low probability in Q; it is less concerned when an unrelated pair ends up too close.

That asymmetry is crucial. t-SNE works hard to keep local friends together, but does not work equally hard to make gaps between distant groups truthful. A gap of 2 centimeters and one of 8 centimeters do not represent a reliable ratio of semantic difference.

Implementations commonly use early exaggeration, temporarily increasing attractive neighborhood probabilities during the initial layout. Afterward the exaggeration is reduced. This is one reason different settings or random starts can produce different-looking islands.

The two coordinates have no intrinsic meaning. Rotating, reflecting, or moving the whole map changes nothing; “the horizontal axis is account risk” is a story imposed by the viewer.

What UMAP changes

UMAP, short for Uniform Manifold Approximation and Projection, starts from a different implementation of the same local question.

It builds a weighted nearest-neighbor graph. Connection weights account for local density, so a distance that is large in a sparse region can be treated differently from the same distance in a dense region. UMAP calls this a fuzzy graph because neighborhood membership is graded.

It then finds a low-dimensional graph with similar connections. Its optimisation uses attractive forces for graph neighbors and repulsive forces for points that should not crowd together. The spring analogy is useful: local edges pull together while unrelated points are pushed apart.

The main controls have clear jobs:

• n_neighbors sets the graph’s scale. A smaller value, such as 15, emphasises very local structure. A larger value, such as 50 or 100, adds context and can join pieces of a broad continuum.
• min_dist controls how tightly points pack in the output. A small value permits compact islands; a larger value spreads points within a group. It changes displayed density, not the true class size.
• metric defines what “near” means in the original space. For text embeddings, cosine is often a sensible starting point.

UMAP is often faster and can transform new points after fitting. It may preserve somewhat more large-scale arrangement than t-SNE on some datasets, but that does not make global distances meaningful. It remains a lossy visual summary.

A practical workflow

Start with the geometry, not the plotting library.

1. Prepare the vectors. Remove obvious duplicates, handle missing values, choose a metric, and scale ordinary numeric columns. Normalise embeddings when cosine similarity is intended. A duplicate-heavy ticket corpus can produce a knot that looks like a category when it is really one template repeated 4,000 times.

2. Consider a first reduction. For thousands of vectors with hundreds or thousands of dimensions, reduce to roughly 30–50 PCA components before t-SNE. This removes some noisy directions and makes pairwise work cheaper. It is a denoising and speed step, not the final visualisation, and it can change the geometry. Compare with a direct run on a smaller sample when that matters. UMAP can often use the original vectors, provided the metric and input are sensible.

3. Choose a starting method. Use UMAP for a quick exploratory map, many points, or a built-in route for placing new points. Use t-SNE for narrowly local questions when slower experimentation is acceptable. Use PCA when you need stable, interpretable, model-ready features rather than a picture.

4. Vary the local scale. For t-SNE, try multiple perplexities, such as 5, 30, and 50. For UMAP, compare at least two n_neighbors values. Treat min_dist as a compactness control, not a cluster detector.

5. Check stability. Run several random seeds and settings. A robust pattern should survive small changes in seed and neighborhood scale. Trustworthiness measures how many plotted neighbors were genuinely close in the original space. A high score does not prove that every island matters, but a poor score indicates that the picture invents many local friendships.

6. Inspect the source data. Read ten points from each apparent island. Look for duplicate templates, customer IDs accidentally embedded in text, language differences, or preprocessing bugs. Labels help inspection; they do not replace it.

The following runnable example uses handwritten digits rather than tickets so it stays self-contained. Each row has 64 pixel features. The sep function measures the fraction of each point’s nearest plotted neighbor with the same digit label. It is a rough diagnostic, not a classifier evaluation.

import numpy as np

from sklearn.manifold import TSNE
from sklearn.decomposition import PCA
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True)   # 64-dim handwritten digits

# PCA to 2D: fast, but classes overlap. t-SNE: slower, classes separate.
pca2 = PCA(n_components=2).fit_transform(X)
tsne2 = TSNE(n_components=2, perplexity=30, random_state=0).fit_transform(X)

# A rough "how separated are the classes" score: mean nearest-neighbor same-label rate
def sep(emb):
    from sklearn.neighbors import NearestNeighbors
    nn = NearestNeighbors(n_neighbors=2).fit(emb)
    idx = nn.kneighbors(emb, return_distance=False)[:, 1]
    return (y[idx] == y).mean()

print(f"PCA   neighbor-purity: {sep(pca2):.2f}")
print(f"t-SNE neighbor-purity: {sep(tsne2):.2f}  <- compare the two")

The code uses labels only after creating the embeddings. The score asks whether nearby plotted points share a known digit label; it does not prove that t-SNE discovered the true generative structure.

The traps you will meet first

The number of islands changes between runs. Four groups with one seed and seven with another may indicate weak evidence, different local optima, or a continuous manifold rather than discrete groups. Compare seeds, perplexities, and n_neighbors, then test any alleged category in the original feature space.

A dense class appears tiny, or a rare class fills half the screen. The display does not preserve population density. t-SNE can expand dense regions and compress sparse ones, while UMAP settings also alter packing. Use source-data counts instead of island area.

Separation disappears when the metric changes. The algorithm may be visualising a bad geometry. Euclidean distance on unnormalised text embeddings, for example, can group tickets by magnitude rather than meaning. Check norms, scaling, duplicates, and the metric before changing model parameters.

The plot is too slow or runs out of memory. Use a representative sample, reduce dimensions with PCA, or use UMAP for a first pass. A sample can form a hypothesis; verify it on the full population before changing a production taxonomy.

Quick check

Next

If you need a broader tour of automated model search, see AutoML. For the linear baseline that these methods often follow, start with PCA.