K-nearest neighbors

A customer opens your support app. You have to predict whether they will churn before anyone has time to call them. There are 600 old customers with known outcomes, but no obvious straight-line rule. Some customers with six monthly visits stayed. Others with six visits left. The useful clue may be which nearby customers behaved similarly.

You could fit a complicated function. Or you could ask a simpler question:

Which old customers most resemble this one?

That is k-nearest neighbors, usually shortened to k-NN. A feature is one measured input, such as monthly visits. A label is the answer you want to predict, such as stay or leave. k-NN stores labeled examples and lets nearby examples vote when a new example arrives.

It has almost no training ceremony. That is both its charm and its trap.

Store, then vote

Suppose a new customer has these two features:

• monthly_visits = 6
• support_tickets = 2

The new row is called the query point: the example waiting for a prediction. k-NN measures its distance to every stored customer, keeps the k shortest distances, and uses their labels.

Here is the complete calculation for the customer example:

With k=3, the neighbors are A, B, and C. Two say stay; one says leave. The prediction is stay.

With k=5, the vote is still 3 stay versus 2 leave. The radius is not fixed: k-NN expands it until it finds the requested number of points. In dense areas that radius is small; in sparse areas it reaches farther.

For regression, k-NN usually averages the neighbors’ target values. Three nearby houses priced at $300,000, $320,000, and $400,000 produce a prediction of $340,000. Distance weighting gives closer neighbors more influence.

The word nearest hides a modeling decision. k-NN needs a distance metric, the rule for deciding how different two rows are. The calculation above uses Euclidean distance: square each feature difference, add the results, then take the square root. Manhattan distance instead adds absolute differences. Different geometries can produce different neighbors and predictions.

Why the vote works

k-NN relies on a local-similarity assumption:

Points close together in feature space tend to have similar labels or target values.

The algorithm does not learn that assumption. You provide the representation and distance rule; k-NN applies them. If “close” means similar customer behavior, it can work beautifully. If numerically close customers are fundamentally different, the vote is irrelevant.

Classification usually uses a uniform vote, where each neighbor counts once. Distance weighting lets a neighbor at distance 0.2 count more than one at distance 2.0. This can help when the closest examples are more informative, but it cannot rescue a bad feature representation.

k controls smoothness

k is a hyperparameter, a setting chosen during model selection rather than learned as a fitted coefficient. It controls how wide each neighborhood is.

At k=1, the prediction follows one example. The boundary can snake around individual observations. On the training data, each row is normally its own closest neighbor, so a 1-NN classifier often gets perfect training accuracy. That is memorization, not evidence of a useful model.

At a much larger k, each prediction averages over a broad region. Local quirks matter less, so the model changes less when the training sample changes: lower variance. But a neighborhood that is too broad mixes genuinely different populations, producing high bias.

In the 600-row churn dataset, k=1 asks what the single most similar customer did. k=500 asks what most of the dataset did. If retention is 90 percent overall, k=500 predicts stay almost everywhere and loses the local signal.

There is no universal best k. It depends on sample size, noise, class balance, feature quality, and the metric. Test several values with cross-validation using only the training data, then evaluate the chosen model once on an untouched test set. Tune the metric and voting scheme too when the data justifies it.

Distance makes scaling part of the model

Distance is arithmetic, not common sense. A feature with large numeric units can drown out every other feature.

Suppose the churn model also includes annual spending. A query customer spends $6,000. Another customer spends $5,900 and differs by one support ticket. Before scaling, the squared distance contribution from spending is 100² = 10,000; the ticket contribution is only 1² = 1. Spending contributes 10,000 times as much to the squared distance, so the algorithm effectively ranks customers by spending.

A common fix is standardization, which converts each numeric feature into roughly “how many training standard deviations from its mean.” The transformation is x' = (x - mean) / standard deviation. A difference of one standard deviation in spending and one in tickets then contribute equally to Euclidean distance.

That is a choice of units, not proof that the features matter equally. If a domain expert says a support ticket should count twice as much as a typical spending deviation, encode that deliberately through feature design or custom weighting.

Fit scaling on the training fold only. Calculating the mean and standard deviation from the entire dataset before cross-validation leaks information from validation rows. Put preprocessing and k-NN in one pipeline so each fold fits its transformer independently.

Scaling does not make irrelevant features useful. Ten useless standardized coordinates can overwhelm two useful ones because Euclidean distance sums contributions from all features. In high dimensions, nearest and farthest points often become surprisingly similar in distance: the curse of dimensionality. Feature selection, dimensionality reduction, or a different model may help.

Categorical data also needs care. Encoding bronze, silver, and gold as 0, 1, and 2 invents an ordered Euclidean distance. One-hot encoding avoids that order but creates different geometry. Mixed data needs a deliberate representation and metric. Feature engineering and encoding is part of k-NN modeling.

A safe tuning pattern

This example keeps the scaler inside the pipeline and chooses k by five-fold cross-validation:

from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(
    n_samples=800,
    n_features=10,
    n_informative=5,
    random_state=0,
)

results = []

for k in [1, 3, 5, 11, 25, 101]:
    model = make_pipeline(
        StandardScaler(),
        KNeighborsClassifier(n_neighbors=k),
    )
    mean_accuracy = cross_val_score(model, X, y, cv=5).mean()
    results.append((k, mean_accuracy))
    print(f"k={k:3d}  CV accuracy={mean_accuracy:.3f}")

best_k = max(results, key=lambda row: row[1])[0]
print(f"\nCV selected k={best_k}")

The scores and winning k depend on the scikit-learn version and generated data. The important behavior is structural: each fold fits StandardScaler on its training portion, then scores the complete pipeline on its validation portion. In a real experiment, reserve a test set before tuning and use it once afterward.

The code uses uniform weighting and Euclidean distance. A real search might include weights="distance" and another suitable metric. Choose a scoring metric that reflects the task: accuracy can look good while missing nearly every rare churn event. See class imbalance and metric selection.

Production reality: easy to fit, expensive to query

k-NN has no learned coefficient vector. Its fitted state is essentially the training features and labels, plus any index structure. Retraining is conceptually simple: add labeled examples and rebuild or update the store.

Prediction is where the cost arrives. A brute-force query compares a new row with every stored row, then keeps the closest k. With n stored rows and d features, that is roughly proportional to n × d distance work per query. Memory also grows with the entire training set. A model that is instant on 10,000 rows can become awkward when every request inspects millions.

Tree-based neighbor indexes help in favorable low-dimensional geometry, while approximate-nearest-neighbor systems trade exactness for lower latency. If latency, memory, or frequent predictions dominate, a compact linear model or tree ensemble may be better; those models do not need every training row at prediction time.

k-NN is strongest when:

• the dataset is small or medium-sized;
• features describe a meaningful, reasonably low-dimensional geometry;
• nearby cases really should have similar outcomes;
• the boundary is irregular rather than usefully linear.

It is a poor choice for high-dimensional noise, raw categorical codes, huge sparse text vectors without a suitable metric, or workloads requiring tiny prediction latency. It also struggles when future cases fall far outside the training population: there may be no trustworthy neighbors.

Failure modes you can see first

Training accuracy is perfect, but validation accuracy is poor. Suspect k=1 or another very small neighborhood: the model is copying examples instead of learning a stable pattern. Tune a wider range of k values with cross-validation and inspect duplicates or contradictory labels.

Almost every prediction is the majority class. A very large k can cause this, but so can class imbalance, a bad metric, or features that place minority examples far away. Check class-specific metrics and inspect the actual neighbors for minority queries. Distance weighting may help a close minority example matter, but it is not a substitute for the right objective.

A unit change changes predictions, or new rows fail. Changing income from dollars to thousands changes the geometry unless preprocessing changes too. Missing values are another common cause; impute inside the pipeline, fit on training data only, and use the same transformation at serving time. Version the preprocessing with the model.

Latency climbs as the customer table grows. The lazy learner is doing more comparisons because it stores more examples. Measure query time at the expected row count, reduce noisy features, use a suitable neighbor index, or switch models when the cost is unacceptable.

For surprising predictions, inspect the neighbors. A k-NN explanation is local: “these rows voted.” If those rows come from a different country, time period, or product line, the model may be answering the wrong question.

Next

k-NN votes from nearby examples. Naive Bayes takes a different shortcut: it estimates class probabilities from feature evidence, often making a fast and surprisingly strong baseline for text.

Quick check