Your t-SNE plot is lying to you (three ways)

At 3:07 p.m., not a.m., your manager points at a t-SNE plot and asks why the red island is twice the size of the blue one.

You have plotted 40,000 support-ticket embeddings. The red points are billing questions: 22,000 tickets. The blue points are shipping questions: 6,000 tickets. On the screen, both islands occupy roughly the same area. A third island, account access, sits far away with a dramatic empty gulf between it and the others.

The tempting story is obvious. Billing must be more varied. Account access must be fundamentally different. The empty space must mean something.

Probably none of those conclusions is justified.

t-SNE and UMAP are excellent instruments for seeing local neighborhoods in data that are otherwise impossible to visualize. They are also unusually good at turning a weak clue into a confident-looking picture. The plot is not broken. The interpretation is.

I have been burned by this in the ordinary engineering way: a beautiful cluster led to a week of taxonomy work, only for the supposed cluster to vanish when the embedding model changed. The useful habit is not to stop using these methods. It is to ask the plot only the question it was built to answer.

First, separate the job from the picture

An embedding is a point represented by a long numeric vector. A language model might represent each support ticket as 768 numbers. Tickets with similar meaning should be near each other according to some distance measure, often cosine distance for text embeddings.

You cannot draw 768 dimensions on a monitor, so you reduce them to two. That reduction is the whole plot. It is not a window onto the data. It is a new arrangement produced by an optimization algorithm.

PCA is the useful baseline. PCA, or principal component analysis, finds straight directions that explain as much variance as possible. If the important structure is curved, folded, or arranged along a nonlinear manifold, a two-dimensional PCA plot can make genuinely similar points overlap.

t-SNE and UMAP take a different approach. They try to keep selected neighbors together rather than preserve the largest directions of global variance. That makes a local group visible. It also means the algorithm is free to distort almost everything else.

This distinction matters. A PCA plot answers something like: “Which two straight directions account for the most variation?” A t-SNE plot answers: “Can I arrange these points so their important local neighbors remain close?” UMAP answers a related question using a graph of approximate neighbors.

Neither question is “What is the true map of the dataset?”

It was not asked to preserve population, inter-island distance, or the shape of the original cloud.

What t-SNE is actually doing

For every point, t-SNE turns nearby points in the original space into probabilities. If ticket A has ten very similar tickets and everything else is far away, those ten receive most of A’s neighborhood probability.

The key setting is perplexity, which is roughly the effective number of neighbors each point is asked to care about. A perplexity of 30 does not mean 30 classes, 30 percent of the dataset, or a radius shared by every point.

It means the local probability distribution behaves approximately as though each point had 30 meaningful neighbors. Dense regions use a smaller physical radius to find them; sparse regions use a larger one.

The algorithm then places points in two dimensions and builds another probability distribution there. It optimizes the arrangement so the two distributions agree. In shorthand, t-SNE minimizes KL(P || Q), where P describes high-dimensional neighborhood affinities and Q describes the proposed two-dimensional affinities.

That direction of the penalty matters. Missing an important neighbor is punished heavily. Adding a few extra neighbors is less costly. Distant pairs in the original space have tiny probabilities and therefore little influence on their exact distance in the final map.

The low-dimensional distribution also uses a heavy-tailed Student t distribution. Its heavier tails give unrelated groups room to sit apart instead of forcing every point into one crowded knot.

This combination produces the characteristic t-SNE picture: compact islands, visible local neighborhoods, and dramatic separation. It is doing exactly what it was asked to do.

It was not asked to preserve population, inter-island distance, or the shape of the original cloud.

The first lie: island size is population

Return to the support tickets. Billing has 22,000 examples. Shipping has 6,000. You see islands of similar apparent area.

That does not mean the classes are equally large. t-SNE adapts its neighborhood radius to local density. A dense region can be spread over a similar visual area to a sparse region because the algorithm is trying to represent local relationships, not draw a proportional map.

A large class may therefore appear as a small, tight island. A small class may appear as a broad island. A class can also split into several islands if it contains several local subtopics.

The same problem affects density. A dark, crowded patch is not automatically a high-density region in the original space. It may simply contain points that the optimization happened to pack closely at the chosen perplexity.

A practical test is to annotate each plotted point with its source count before interpreting the image. If the visual story conflicts with the table, trust the table.

The second lie: the gap is a measurement

Suppose the account-access island is 400 pixels from the billing island, while shipping is only 80 pixels away.

It is tempting to say account-access tickets are five times more different from billing tickets. The coordinates do not support that arithmetic.

Once t-SNE has made two groups sufficiently separate, the exact distance between them has little value. Many arrangements can preserve the same local neighborhoods. The optimizer can move one island farther away without substantially damaging the objective, because most cross-island pairs already have very small affinity.

Even the direction can be arbitrary. Rotate the entire plot and nothing changes. More subtly, a different seed or perplexity can move, rotate, stretch, or reorder islands while preserving much of the local structure.

UMAP is not a cure for this. Its graph construction can make broad connectivity easier to inspect, especially with a larger n_neighbors setting, but the coordinates are still the result of a two-dimensional optimization. A long bridge in a UMAP plot may reflect a real chain of neighboring points.

The distance across two disconnected-looking components is still not a calibrated measure of semantic difference.

If you need to know whether account access is genuinely farther from billing than shipping is, measure distances in the original embedding space. Use the metric that matches the representation, compare distributions rather than cherry-picking two points, and report uncertainty.

The 2-D plot is a useful prompt for that analysis, not a replacement for it.

The third lie: shape and stability are ground truth

A t-SNE island shaped like a crescent looks meaningful. A narrow bridge looks like a transition. A hole looks like a missing category.

Sometimes those patterns are real. They are not automatically real because they look geometric.

The shape depends on the neighborhood scale. With 40,000 tickets, perplexity 10 emphasizes tiny local pockets. Perplexity 100 asks each point to participate in a broader neighborhood.

A group that looks like one smooth arc at one setting may become five compact islands at another. Both plots can be internally coherent because they are answering different local questions.

Initialization and randomness matter too. t-SNE starts from an initial arrangement and uses iterative optimization. Change the random seed and some local relationships may remain while the large visual composition changes.

PCA initialization can make results more repeatable in some workflows, but it does not turn t-SNE into a globally faithful map.

This is why a single plot is weak evidence. A pattern that survives several seeds, perplexities, samples, and embedding versions deserves investigation. It still does not prove that the data contains a natural cluster.

Validate the visual groups

A common failure shows up later: the plot has three gorgeous islands, but a classifier trained on the original embeddings performs poorly on held-out tickets. That result is useful only after you define the target labels, the outcomes you actually want to predict—for example, human-reviewed intent labels rather than the three colors someone drew on the plot.

Evaluate a baseline appropriate to that target and representation, using a held-out split that resembles the tickets arriving in production. A regularized linear classifier is a sensible first check for a roughly linear boundary; a plausible nonlinear boundary needs a suitable nonlinear baseline too.

Poor held-out performance means the plot has not established a decision boundary that generalizes to that target. It does not prove that t-SNE created the islands.

A linear model can miss a genuine nonlinear grouping, labels can be noisy or fail to match the visual groups, and sample shift can make a previously useful pattern fail on future tickets.

If the groups are unsupervised, there are no target labels to classify. Cluster the original embeddings instead, then check whether the clusters remain similar across:

• resamples;
• distance metrics;
• preprocessing choices; and
• clustering settings.

UMAP wins on scale, not on truth

UMAP usually becomes the practical first choice when the dataset is large. The reason is structural.

A standard exact t-SNE calculation considers interactions among many pairs of points. Barnes–Hut and other approximations reduce that burden, but t-SNE can still become expensive as the number of points and optimization steps grow.

UMAP first builds a sparse nearest-neighbor graph, often using approximate nearest-neighbor search. It then optimizes relationships represented by that graph rather than maintaining a dense set of all pairwise interactions.

The graph and optimization are still substantial, but they generally scale more comfortably.

That makes UMAP a good choice for a million-row exploratory sample where t-SNE may be impractical. Do not turn that into a universal speed promise. Wall-clock time depends on the implementation, hardware, metric, dimensions, graph settings, and whether you are plotting every point or a sample.

UMAP also exposes useful controls:

• n_neighbors controls how broad the graph’s notion of neighborhood is. Smaller values emphasize local pockets; larger values make broader connectivity more visible.
• min_dist controls how tightly points are allowed or encouraged to pack in the final representation. Smaller values create denser-looking clumps.
• The distance metric defines what “near” means before the graph is built.

Those settings change the question. They do not reveal a hidden canonical answer.

UMAP’s reputation for preserving global structure is also easy to overstate. It can show more of the connectivity between local regions than a typical t-SNE run, but global distances and cluster areas remain objective-dependent.

Treat “better global structure” as a useful tendency, not a guarantee.

The strongest argument for trusting the plot

The fair counterargument is this: nobody serious claims a t-SNE plot is a geographic map. The plot is exploratory. If it consistently reveals clean groups that PCA hides, why be so fussy?

Because exploration still creates decisions. Someone chooses the embedding model, distance metric, sample, seed, perplexity, color scheme, and the screenshot that goes into the presentation.

A pattern that appears only after twelve tries is not a discovery; it is a selection effect wearing a lab coat.

The answer is not to demand proof from an exploratory visualization. It is to label the result correctly.

“Tickets form three stable local neighborhoods under these settings” is a defensible observation.

“Account-access tickets are a separate population, five times farther from billing, and should get their own product team” is a chain of unsupported conclusions.

What to do on Monday morning

Start by freezing the input. Record:

• the embedding model and version;
• preprocessing;
• distance metric;
• sample-selection rule; and
• timestamp.

If your embeddings came from a changing language-model pipeline, a new model can move the points before t-SNE or UMAP has done anything wrong.

Make a PCA view first. For a 768-dimensional embedding, reducing to 50 dimensions before t-SNE can reduce noise and computation, but it is a preprocessing choice, not a claim that 50 dimensions contain everything. Keep the original vectors available for verification.

Then run a small grid rather than one favorite setting:

• For t-SNE, compare at least three perplexities, such as 15, 30, and 100, provided they are sensible for your sample size.
• Run each setting with several seeds.
• For UMAP, compare local and broader graphs, such as n_neighbors=15, 50, and 100.
• Try more than one min_dist when the apparent compactness matters.

Save the coordinates with the full metadata. Do not save only the screenshot.

Plot the same points in several ways:

• with class labels;
• with no labels; and
• colored by batch, source system, date, or annotator.

If a “semantic cluster” is actually one export job from Tuesday, you want to find that before the product review.

Verify before acting

For every interesting group, verify the claim in the original space. Inspect nearest neighbors using the original vectors. Compare within-group and between-group distances there.

If there is a defined target label, evaluate an appropriate baseline on the original embeddings with a representative held-out split. Poor held-out performance means the visual separation has not established a reliable decision boundary for that target; it does not, by itself, show that t-SNE created an artifact.

If the groups are unsupervised, cluster the original embeddings and check cluster stability across:

• resamples;
• preprocessing choices;
• distance metrics; and
• clustering settings.

Do this instead of turning plot colors into labels.

You can also measure neighborhood preservation with metrics such as trustworthiness. These compare nearest-neighbor relationships before and after projection. They are not a certificate that the map is globally true, but they are better than judging every result by how aesthetically pleasing the islands look.

Finally, write the plot’s settings in its caption:

“UMAP, cosine metric, 40,000 tickets, n_neighbors=50, min_dist=0.1, seed 7, embeddings from model version X.”

That sentence prevents a surprising amount of future archaeology.

When to use something else

Use t-SNE when local neighborhood inspection is the main question and the dataset is manageable. Use UMAP when you need that style of inspection at larger scale or want to experiment with neighborhood breadth.

Use PCA when you need a deterministic, cheap transform that can be applied consistently to new data. It is often a sensible first reduction before a nonlinear visualization. It is also easier to put into a production pipeline because its learned projection is explicit and stable.

Do not feed a 2-D t-SNE plot into a downstream model just because the colors separate nicely. The projection discards information, depends on visualization settings, and may not provide a clean transform for new examples.

UMAP can support transformations in some implementations, but that does not make a 2-D visualization a sound feature representation. If you want reduced features, choose the dimension and method for the predictive task, then validate it with model-selection discipline, not with a convincing screenshot.

A good t-SNE or UMAP plot should leave you with better questions:

Which tickets are nearest in the original space? Does the grouping survive a new sample? Is the apparent boundary predictive? Is the “outlier” a rare case, a mislabeled row, or a broken embedding?

The plot earns its keep when it sends you to those checks. It lies when you let it answer them.

For the broader craft of reading model-generated visualizations without mistaking appearance for evidence, ML plots is the useful companion.