Gaussian mixture models

K-means forces every point into exactly one cluster and assumes those clusters are round, equal-sized blobs. Gaussian mixture models (GMMs) relax both: they model the data as a blend of Gaussian “bells,” give each point a probability of belonging to each one, and allow elongated, differently shaped clusters. GMM is the probabilistic generalization of k-means.

Soft assignment — the key difference

K-means says “point 7 is in cluster B.” A GMM says “point 7 is 70% cluster B, 30% cluster A.” For points deep inside a cluster that distinction barely matters — but for points in the overlap between clusters, the soft probability is far more honest than a coin-flip hard label.

GMMs also fit covariance per cluster — so clusters can be stretched, rotated ellipses, not just circles. That alone lets them fit data k-means mangles.

Fitting with EM

You can’t directly compute the clusters because you don’t know which point came from which Gaussian. The Expectation-Maximization (EM) algorithm solves this by alternating, much like k-means does:

1. E-step — given the current Gaussians, compute each point’s probability of belonging to each (soft assignment).
2. M-step — given those probabilities, update each Gaussian’s mean, covariance, and weight.

Repeat until it converges. In fact, k-means is a special case of EM with hard assignments and fixed spherical covariance — GMM is the fuller, probabilistic version.

import numpy as np
from sklearn.mixture import GaussianMixture
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

# Stretched, overlapping clusters that k-means struggles with.
X, _ = make_blobs(n_samples=500, centers=2, cluster_std=[1.0, 2.8], random_state=2)

gmm = GaussianMixture(n_components=2, covariance_type="full", random_state=0).fit(X)
proba = gmm.predict_proba(X)            # soft: probability per cluster

# Find the most uncertain point (closest to a 50/50 split)
uncertain = np.argmin(np.abs(proba[:, 0] - 0.5))
print(f"Most ambiguous point: {proba[uncertain].round(2)} (cluster probabilities)")
print(f"GMM can express '{proba[uncertain][0]:.0%} A, {proba[uncertain][1]:.0%} B' —")
print("k-means would force it fully into one cluster.")
print(f"\nModel selection: BIC = {gmm.bic(X):.0f} (try a few n_components, pick the lowest BIC)")

In one breath

• A GMM models data as a blend of Gaussian “bells” and gives each point a probability of belonging to each — soft assignment, where k-means forces a hard label.
• It also fits a covariance per cluster, so clusters can be stretched, rotated ellipses, not just circles.
• It’s fit by EM: the E-step computes soft memberships, the M-step updates each Gaussian’s mean, covariance, and weight, repeated to convergence.
• k-means is the special case of EM with hard assignments and fixed spherical covariance.
• Choose the number of components with BIC/AIC (lower is better), use several restarts to dodge local optima, and switch to DBSCAN for non-Gaussian shapes.

Quick check

Next

That completes the clustering family. To visualize any of these clusters in 2D, project with t-SNE & UMAP.