Why shouldn't you use t-SNE output as features for a downstream model, and what would you use instead?

How to think about it

The crisp answer

When a downstream model, meaning the later predictor that consumes these features, must score tomorrow’s data, ordinary t-SNE coordinates are the wrong handoff. I would use t-SNE to inspect the data, not as production features: its 2D or 3D output optimizes local visual neighborhoods rather than a stable predictive representation, and classic t-SNE has no dependable transform for a new row.

At 3 a.m., the practical problem is simple: training produced coordinates, but serving has no principled way to produce compatible coordinates for the next customer.

Why t-SNE breaks as a feature extractor

The interviewer is testing whether you understand what t-SNE optimizes.

For every pair of high-dimensional points, t-SNE converts their similarity into a probability. Nearby points get a large probability; distant points get a small one. It then searches for low-dimensional coordinates whose probabilities resemble the original ones. The loss is a form of KL(P || Q), called Kullback–Leibler divergence, which measures how different the original similarities P are from the low-dimensional similarities Q.

That loss is deliberately local. If two points were close in the original space but end up far apart, the penalty is large. If two points were far apart but happen to land near each other, the penalty is much less important. The algorithm therefore spends its limited 2D space protecting neighborhoods, not preserving the full map.

This is why a t-SNE plot can show three attractive islands while telling you almost nothing reliable about the distance between those islands. A point at coordinate (4, 2) is not meaningfully “twice as far” from a point at (2, 1) than another point is. Cluster diameter, empty space, and global ordering are not quantities the objective promises to preserve.

The coordinate axes themselves have no useful semantics. A different random seed, perplexity, initialization, or optimization path can rotate, reflect, stretch, or rearrange the picture while preserving much of the same neighborhood structure. Perplexity is roughly the local neighborhood size t-SNE pays attention to; it is not the number of clusters. Changing it from 30 to 5 asks a different question of the data.

There is also a more basic engineering problem. Standard t-SNE jointly places all points in the dataset. It does not naturally learn a reusable function such as “take any new row x and return its coordinates.” A new row can change the relationships that determine the existing layout. Fitting a separate t-SNE run for test data creates a different coordinate system, so the downstream model’s learned boundary no longer has a dependable meaning.

A concrete train/test failure

Suppose a credit-risk team has 60,000 historical applications with 40 standardized features. The team holds out 20,000 later applications for testing and wants to predict default.

An engineer runs t-SNE with n_components=2 and trains logistic regression on the 60,000 two-dimensional coordinates. The training code is straightforward:

from sklearn.manifold import TSNE

tsne = TSNE(
    n_components=2,
    perplexity=30,
    init="pca",
    random_state=7,
)

z_train = tsne.fit_transform(X_train)

# There is no standard tsne.transform(X_test) in scikit-learn.

The first symptom may be an error such as AttributeError: 'TSNE' object has no attribute 'transform'.

There are three tempting workarounds, and each has a problem.

1. Fit t-SNE on all 80,000 rows before splitting.
   The test rows influence the similarities and therefore the training layout. No labels were used, so this is not label leakage, but it is still test-set contamination in an ordinary evaluation. The setup no longer matches production, where future applications are not available when the representation is fitted.

2. Fit one t-SNE on the training rows and another on the test rows.
   The second run may rotate the map, change cluster spacing, or place neighborhoods differently. A classifier trained on the first coordinate system cannot safely interpret the second.

3. Place each test row using a nearest-neighbor hack.
   That can be built, but it is now a separate approximation that must be specified, versioned, and validated. It does not make t-SNE distances predictive or solve the instability of the representation.

A nice plot of the 60,000 training applications does not rescue the design. t-SNE may separate applications by region, product type, or data-collection process even when those groups are weak predictors of default.

What I would use instead

The choice depends on why compression is needed.

Need	Reasonable choice	What it gives you
Stable linear compression	PCA	A fitted projection that can transform new rows
Nonlinear compression	Autoencoder	A reusable learned encoder
Representation tuned to the target	Supervised encoder	Features trained for the actual prediction task
No serious dimensionality problem	Original features with regularization	No unnecessary information loss
A manifold plot that must accept new rows	UMAP, cautiously	Common implementations provide a transform, but geometry still needs validation

PCA for a dependable baseline

Principal component analysis, or PCA, learns directions that capture as much variance as possible. If μ is the training mean and W_k contains the first k learned directions, a new row x is transformed as z = (x - μ) W_k.

That equation is the important contrast with ordinary t-SNE. The mean and directions are fitted once on training data and can be applied to validation, test, and production rows consistently.

For the credit example, I might evaluate eight PCA components like this:

from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

model = make_pipeline(
    StandardScaler(),
    PCA(n_components=8),
    LogisticRegression(max_iter=1000),
)

model.fit(X_train, y_train)
test_probability = model.predict_proba(X_test)[:, 1]

The pipeline fits scaling and PCA only on X_train, then applies the saved transformations to X_test. In production, I would persist that fitted pipeline, not refit it on each request.

PCA is not automatically better for prediction. It preserves variance, not necessarily information about default. A rare but highly predictive feature can contribute little total variance and be discarded by an aggressive PCA bottleneck. I would compare PCA against the original 40 features with regularization and choose the number of components using validation performance.

Autoencoders for nonlinear structure

An autoencoder is a neural network with an encoder that compresses an input and a decoder that reconstructs it. The compressed vector is the latent representation. At inference time, the encoder is a reusable function, so it can process a new application.

This is useful when the data has nonlinear structure that PCA cannot capture. The trade-off is real: the autoencoder learns to reconstruct the input, not necessarily to predict default. It may preserve a customer’s spending pattern while discarding a small signal that matters for risk. It also needs more data, tuning, monitoring, and protection against overfitting than PCA.

Supervised representations when the target matters

If the goal is default prediction, a supervised encoder can learn representations while optimizing a loss that includes the default label. A neural classifier’s hidden layer, a task-specific embedding, or another supervised dimensionality-reduction method may be appropriate.

This aligns the representation with the business objective, but it introduces target leakage risks and makes validation more important. The encoder must be fitted only with information available at prediction time. If the target or a future-derived feature sneaks into representation training, the resulting score can look excellent right up until the first real batch arrives.

The senior-level nuance

I would not say “t-SNE can never be used as a feature.” If the dataset is fixed, every row is available at once, no new rows will ever arrive, and the downstream task is explicitly transductive, t-SNE coordinates can be tested as candidate features. For example, a one-off analysis of a fixed image collection might use them in an exploratory classifier.

That is a narrow exception, not a production recommendation. I would still test several seeds and perplexities, use a held-out protocol, and compare against simple baselines. A method can be useful for a fixed benchmark while being unsuitable for a deployed model.

Some modern libraries expose an approximate transform for UMAP or offer parametric versions of t-SNE. That solves the mechanical problem of placing new points, but not the statistical one. A transform API does not make global distances meaningful or prove that the representation contains stable signal. The downstream metric must improve across seeds, time splits, and genuinely unseen data.

UMAP is therefore a possible engineering choice for a validated embedding, especially when a new-point transform matters. It is not a magic “t-SNE, but safe” button.

What failure looks like in practice

A common symptom is unstable validation. With one seed, the classifier appears to find a useful boundary; with another, the AUC changes sharply even though the source data did not. The plot still looks convincing, which makes this failure especially expensive.

Another symptom is that a model works on the original t-SNE plot but cannot score test data without refitting the embedding. If someone proposes fitting t-SNE separately for every batch, that is the moment to stop. The model is no longer consuming a consistent feature definition.

What they’ll ask next

“Is fitting t-SNE without labels still leakage?”

It avoids label leakage, but it can still contaminate evaluation if test rows influence the fitted representation. More importantly, unsupervised fitting does not fix the lack of a stable out-of-sample mapping. I would fit preprocessing on the training partition and apply a reusable transform to the test partition.

“Does UMAP solve the problem?”

It improves the engineering story because common UMAP implementations can transform new points after fitting. But UMAP is still primarily a neighborhood and manifold visualization method. Its coordinates remain sensitive to settings, and global distances are not automatically predictive. I would use it downstream only after validating the complete train-transform-test pipeline.

“Why not always use PCA?”

Because PCA optimizes variance, not the target. If the original features are manageable and a regularized model performs well, compression adds risk without benefit. If PCA drops useful nonlinear or low-variance signal, an autoencoder or supervised representation may be better. The answer comes from a validation comparison, not from the appearance of a 2D plot.

Say this in the interview

“t-SNE is for seeing local neighborhoods, not for learning a stable feature map; I would use train-fitted PCA for linear compression, a validated encoder for nonlinear or supervised features, and keep the original inputs when compression does not improve validation performance.”