Bag-of-words & TF-IDF

A model can’t multiply "cat". Once you have clean tokens, the next step is vectorization — turning a document into numbers. The oldest idea is also the most durable: bag-of-words. Represent each document as a vector of word counts, one dimension per vocabulary word. It’s called a bag because you toss the words in and throw away the order — "dog bites man" and "man bites dog" get the identical vector.

Bag-of-words, and its flaw

Build a vocabulary from the corpus, then each document becomes a count over that vocabulary — a long, mostly-zero (sparse) vector. Simple and surprisingly useful. But raw counts have a flaw: the most frequent words are the least informative. the might be the highest count in every document and tell you nothing about what the document is about. We want to reward words that are frequent in this document but rare across the collection — distinctive words. That’s exactly TF-IDF:

• Term frequency (TF) — how often the word appears in this document (its count, often normalized by document length).
• Inverse document frequency (IDF) — log(N / df), where df is how many of the N documents contain the word. Common-everywhere words have high df, so low IDF; rare words have high IDF.
• TF-IDF = TF × IDF — high only when a word is frequent here and rare elsewhere.

import math

corpus = ["the cat sat on the mat", "the dog sat on the log", "cats and dogs are pets"]
docs = [d.split() for d in corpus]
N = len(docs)

def tf(w, d):  return d.count(w) / len(d)
def idf(w):    return math.log(N / sum(w in d for d in docs))

d0 = docs[0]
print(f"doc0 = {corpus[0]!r}   (corpus of {N} docs)")
for w in ["the", "cat", "mat"]:
    print(f"  {w:<4} tf={tf(w, d0):.3f}  idf={idf(w):.3f}  tfidf={tf(w, d0) * idf(w):.3f}")

doc0 = 'the cat sat on the mat'   (corpus of 3 docs)
  the  tf=0.333  idf=0.405  tfidf=0.135
  cat  tf=0.167  idf=1.099  tfidf=0.183
  mat  tf=0.167  idf=1.099  tfidf=0.183

Look at what happened: the is the most frequent word in the document (tf 0.333, twice the count of the others) yet ends with the lowest TF-IDF, because it appears in 2 of 3 documents (low IDF). cat and mat, each appearing once but only in this document, score higher. TF-IDF has automatically discovered which words actually characterize the document.

Why it endures — and where it stops

TF-IDF is the workhorse behind classic search and information retrieval: rank documents by the TF-IDF overlap with the query, and you have a serviceable search engine (the industry standard BM25 is a refined TF-IDF with length normalization and saturation). The same vectors feed text classification (Naive Bayes, SVMs) and similarity (cosine between TF-IDF vectors). It’s fast, interpretable, needs no training, and is a baseline you have to beat.

But the bag has hard limits, all stemming from what it throws away:

• No word order — “not good” and “good not” are identical; negation and syntax vanish.
• No meaning — car and automobile are different dimensions with zero similarity; the representation is blind to synonyms.
• Huge and sparse — one dimension per vocabulary word (tens of thousands), almost all zero.

In one breath

• Bag-of-words represents a document as a vector of word counts over the vocabulary — sparse, high-dimensional, and order-free (“dog bites man” = “man bites dog”).
• Raw counts over-weight ubiquitous words; TF-IDF fixes it: TF (frequency in this doc) × IDF (log(N/df), low for everywhere-words, high for rare ones).
• So a word scores high only when it’s frequent here and rare elsewhere — the demo’s the is most frequent yet lowest TF-IDF, while distinctive cat/mat win.
• It powers search/IR (BM25 is refined TF-IDF), text classification (Naive Bayes/SVM), and cosine similarity — fast, interpretable, training-free.
• Its limits — no order, no meaning (synonyms), huge sparse vectors — motivate dense embeddings; but TF-IDF/BM25 remain a strong baseline and the keyword half of hybrid search.

Quick check

Next

TF-IDF’s blind spot — no sense of meaning — is exactly what word embeddings fix by learning a dense vector space; see embeddings. These vectors also feed classic text classifiers like Naive Bayes.