How do you choose k in k-means, and when does k-means fail?

How to think about it

The crisp answer

There is no universally correct k, so I choose it by combining the WCSS elbow, silhouette score, repeated-run stability, and domain usefulness. K-means fails when its Euclidean, squared-distance geometry does not match the data: for example, non-convex shapes, very different densities, influential outliers, mixed categorical data, or a problem with no meaningful groups at all.

Why the choice matters

K-means represents each cluster with a centroid, which is the coordinate-wise mean of the points assigned to that cluster. Its objective is to minimize the within-cluster sum of squares, or WCSS: the total squared distance from every point to its cluster centroid.

In plain notation:

J = sum of squared distances from each point to its assigned centroid

The algorithm, usually Lloyd’s algorithm, alternates between two steps:

1. Assign every point to its nearest centroid using Euclidean distance, meaning ordinary straight-line distance.
2. Recalculate each centroid as the mean of the points assigned to it.

It repeats until the assignments stop changing or the improvement becomes negligible.

That mechanism explains both the usefulness and the limitations. Squared distance rewards compact groups. It also makes distant points disproportionately expensive. And because each point is assigned to its nearest centroid, the resulting boundary between two clusters is a straight line, or a flat hyperplane in higher dimensions.

So k-means naturally produces compact, convex regions. Convex means a shape without inward dents: if you draw a line between any two points in the group, the line stays inside the group. A crescent or ring is not shaped that way.

“Similarly sized and spherical clusters” is a useful rule of thumb, not a strict mathematical requirement. K-means can handle some unequal clusters. It becomes unreliable when the unequal size or spread makes a different partition cheaper under its squared-distance objective.

How I choose k

I start with the data, not the metric.

K-means is designed for numeric features where averages and straight-line distance make sense. I remove identifiers and accidental leakage, decide which features should influence similarity, and usually standardize each numeric feature. Standardization subtracts a feature’s mean and divides by its standard deviation, putting features on comparable scales.

This matters because distance has no idea what a feature means. Suppose one feature is annual spend from zero to 100,000 dollars and another is weekly logins from zero to 20. Without scaling, spend will dominate almost every distance. The resulting clusters may be “high spend versus low spend,” even if the product team wanted engagement to matter equally.

I then fit several candidate values, often a plausible range such as k = 2 through k = 10. For each value, I use k-means++ initialization and multiple restarts. The n_init parameter controls how many initializations are tried; the implementation keeps the run with the lowest WCSS.

A minimal scikit-learn pattern is:

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)

results = []
for k in range(2, 11):
    model = KMeans(
        n_clusters=k,
        init="k-means++",
        n_init=20,
        random_state=0,
    )
    labels = model.fit_predict(X_scaled)
    results.append(
        {
            "k": k,
            "wcss": model.inertia_,
            "silhouette": silhouette_score(X_scaled, labels),
        }
    )

Here, inertia_ is scikit-learn’s name for the WCSS objective.

The elbow method

The elbow method plots WCSS against k. WCSS always decreases as k increases, because giving the algorithm more centroids cannot make the best possible fit worse. At k equal to the number of rows, every point can have its own centroid and WCSS becomes zero. That is not a useful model. It is memorization wearing a lab coat.

The goal is to find where extra clusters stop buying much compactness.

Consider six subscription accounts with a single preprocessed risk score:

[1, 2, 3, 10, 11, 12]

The best WCSS values for a few choices are:

k	WCSS	Improvement from previous k
1	125.5	—
2	4.0	121.5
3	2.5	1.5
4	1.0	1.5

The large drop from one cluster to two tells us that two groups explain most of the structure. Splitting further helps much less. The elbow points to k = 2.

The elbow is not a theorem. Some datasets have a smooth curve with no visible bend. In that case, pretending to see one is less scientific than admitting the plot is undecided.

The silhouette score

The silhouette score measures whether a point is closer to its own cluster than to the nearest alternative cluster. For point i:

• a(i) is its average distance to points in its own cluster.
• b(i) is the smallest average distance to points in another cluster.
• s(i) = (b(i) - a(i)) / max(a(i), b(i))

The score ranges from negative one to one. A value near one means the point is much closer to its own cluster. A value near zero means it sits on a boundary. A negative value suggests it may fit another cluster better.

For the six scores above, the two natural groups have an average silhouette of about 0.85, because the within-group distances are small and the gap between groups is large.

Silhouette is useful, but it is not a judge handed down from the clustering gods. It prefers compact, well-separated groups and can favor a small k. It can also penalize a legitimate small cluster or behave strangely when densities differ. I use it as evidence, not as an automatic answer.

Stability and usefulness

I also rerun the model with different seeds and on bootstrap samples or time slices. A useful clustering should not rearrange dramatically because one centroid started 10 centimeters away.

To compare runs, I use a partition metric such as the Adjusted Rand Index, or ARI, which measures agreement between two clusterings while correcting for agreement expected by chance. I do not compare raw cluster numbers directly: cluster 0 in one run may be cluster 2 in another even when the partitions are identical.

Finally, I ask whether the groups support a decision. Suppose k = 4 has a silhouette of 0.47 and k = 5 has 0.49, but the fifth cluster contains only 1.5 percent of customers and receives the same treatment as another group. The extra cluster may add geometric detail without adding business value. If the marketing team can operate three distinct campaigns, k = 3 may be the better choice.

A good answer is therefore not “the silhouette picked five.” It is: “Five was slightly better geometrically, but four was stable, interpretable, and actionable, so I chose four.”

When k-means fails

The failure usually comes from a mismatch between the data’s geometry and the objective being optimized.

Data or problem	What you observe	Why k-means struggles
Two crescents or concentric rings	Each true shape gets cut into pieces	Nearest-centroid boundaries are straight, not curved
Very different densities	A dense group is absorbed or a diffuse group is split	Reducing many squared errors in the large group can outweigh preserving the small one
Outliers	A centroid moves toward an extreme point	Squaring distance gives far-away points high leverage
Categorical features	Centroids such as “half blue, half red” have no meaning	Means and Euclidean distance may not represent similarity
A continuous gradient	The chosen groups change as k changes	There may be no natural partition to discover
High-dimensional sparse data	Silhouette is weak and distances look similar	Distances can lose contrast as dimensions increase

For non-convex shapes, spectral clustering or a density-based method such as DBSCAN may be a better starting point. DBSCAN can label noise and find irregular shapes because it groups points by local density rather than by distance to a mean. It has its own weakness: one global density setting is difficult when the data contains both dense and sparse groups.

For elliptical clusters and soft membership, a Gaussian mixture model is often more appropriate. It models each group as a probability distribution, so a point can belong to cluster A with probability 0.7 and cluster B with probability 0.3. That is useful when boundaries are genuinely uncertain.

For outlier-heavy numeric data, k-medoids is worth considering. A medoid is an actual data point, so it is less vulnerable than a mean to an extreme observation. For mixed data types, I would choose a distance designed for those types, then use a compatible clustering method rather than forcing everything into ordinary Euclidean k-means.

Common trap: more restarts do not repair the wrong model. k-means++ spreads initial centroids more intelligently, and n_init=20 gives twenty chances to find a better local solution. Neither can turn two rings into two rings, because the objective still wants centroid-based convex regions.

A production failure you can spot early

Imagine a customer-segmentation job retrained every Monday. One week, “cluster 2” contains 18 percent of customers; the next week, it contains 3 percent. The centroids move substantially, and a customer who has not changed behavior switches segments.

First check label permutation: the integer labels may simply have been renamed. Compare partitions with ARI and inspect centroids in the original business units.

If the partitions genuinely differ, likely causes include unstable initialization, too many clusters, a small dataset, changing feature distributions, or a weak cluster structure. Increase restarts, test stability on time windows, reconsider the features, and compare another algorithm. Do not silently publish a new campaign assignment because the code completed successfully.

When scoring a new customer, k-means will always assign that customer to some cluster, even if the customer is unlike every training example. I would monitor the distance to the nearest centroid as an out-of-distribution signal, along with cluster sizes and feature distributions. A label alone can make an unfamiliar customer look reassuringly classified.

For a deeper treatment of the algorithm and its geometry, see k-means clustering.

What they’ll ask next

“Does the silhouette score automatically choose k?”
No. It ranks candidate partitions according to compactness and separation under the chosen distance. It cannot tell whether the clusters are useful, stable over time, or meaningful to the business. It can also prefer a small k or penalize unequal-density structure.

“What do k-means++ and n_init solve?”
K-means++ chooses initial centroids that are spread out, with later centers sampled with probability related to squared distance from existing centers. n_init runs the algorithm from multiple initializations and keeps the lowest-WCSS result. Together they reduce sensitivity to initialization; they do not solve feature scaling, outliers, the wrong k, or non-convex shapes.

“What would you use instead?”
I would choose based on the failure mode: DBSCAN for irregular shapes and noise, a Gaussian mixture for elliptical clusters and soft assignments, k-medoids for outlier-prone data, and a method built around an appropriate distance for categorical or mixed data. If no method gives stable, actionable groups, I would not force a clustering solution.

Say this in the interview

“I choose k by triangulating the WCSS elbow, silhouette, stability across restarts and samples, and whether the resulting groups support a real decision; I avoid k-means when the data is non-convex, badly scaled, outlier-heavy, categorical, or does not contain meaningful centroid-shaped clusters.”