Topic modeling & LDA

You’re handed ten thousand news articles with no categories. What are they about? Reading them is impossible; you want the themes to emerge on their own. That’s topic modeling — unsupervised discovery of the latent topics running through a collection. The classic method is Latent Dirichlet Allocation (LDA), and it rests on two simple ideas.

Two ideas: word distributions and topic mixtures

LDA models a corpus with two kinds of distribution:

• A topic is a distribution over words. The “animals” topic puts high probability on cat, dog, pet; the “finance” topic on bank, money, loan. A topic is its characteristic word weights — no name attached (you read the top words and name it yourself).
• A document is a mixture of topics. An article isn’t one topic; it’s 90% animals, 10% finance. Each document has its own blend.

This is a generative story: to write a document, first draw its topic mixture, then for each word position draw a topic from that mixture and a word from that topic. Of course we don’t watch anyone write — we only see the finished words. LDA runs the story backward: given the observed words, infer the hidden topic-word distributions and per-document mixtures that most plausibly produced them. With the topics known, assigning a document is just asking which topics explain its words:

import math

topics = {  # what LDA learns: each topic is a distribution over words
    "animals": {"cat": 0.4, "dog": 0.4, "pet": 0.2},
    "finance": {"bank": 0.5, "money": 0.3, "loan": 0.2},
}

def topic_scores(doc):  # log-likelihood of the doc's words under each topic
    return {name: sum(math.log(dist.get(w, 1e-6)) for w in doc) for name, dist in topics.items()}

for doc in [["cat", "dog", "pet"], ["bank", "loan", "money"], ["cat", "bank"]]:
    s = topic_scores(doc); best = max(s, key=s.get)
    print(f"{doc} -> {best:<8} (animals={s['animals']:.1f}, finance={s['finance']:.1f})")

['cat', 'dog', 'pet'] -> animals  (animals=-3.4, finance=-41.4)
['bank', 'loan', 'money'] -> finance  (animals=-41.4, finance=-3.5)
['cat', 'bank'] -> finance  (animals=-14.7, finance=-14.5)

The pure documents land firmly in one topic; the mixed ['cat', 'bank'] is nearly tied (−14.7 vs −14.5) — exactly the case where LDA would report a blended mixture rather than forcing a single label.

The Dirichlet in the name is the prior that keeps things sparse — it nudges each document toward few topics and each topic toward few words, which is what makes the results interpretable. Inference is done with Gibbs sampling or variational methods, and you choose the number of topics K up front.

Topic modeling vs clustering

It looks like clustering, but with a crucial difference: clustering assigns each document to one cluster (hard), while LDA gives each document a mixture — soft, overlapping membership. A document can be 60% politics and 40% economics at once, which is truer to how real documents work. And LDA’s topics are interpretable by their top words, whereas a geometric cluster is just a centroid.

In one breath

• Topic modeling discovers latent themes in an unlabeled corpus; LDA is the classic.
• Two ideas: a topic is a distribution over words (animals: cat/dog/pet), and a document is a mixture of topics (90% animals, 10% finance).
• It’s a generative story (draw a mixture, then per word draw a topic then a word) run backward — infer the hidden topics and mixtures from the observed words (the demo’s mixed [cat, bank] comes out nearly tied).
• The Dirichlet prior enforces sparsity (few topics per doc, few words per topic) for interpretability; inference via Gibbs/variational, and you pick K.
• Unlike clustering (one hard cluster per doc), LDA gives soft, overlapping mixtures and interpretable topics; modern embedding-based methods (BERTopic) capture meaning it can’t.

Quick check

Next

LDA organizes a corpus by theme; to retrieve from it by query, see information retrieval & BM25. Its hard-assignment cousin is clustering, and the embeddings behind modern topic models are embeddings as vectors.