What is TF-IDF and how does it improve on raw bag-of-words counts?

How to think about it

Suppose a search query is the cat. A raw count vector can make the, which appears in nearly every document, look as useful as cat. TF-IDF improves on this by multiplying a term’s frequency inside a document by its rarity across the whole collection, so frequent and distinctive terms carry more weight than frequent and generic ones.

Why raw counts mislead

A bag-of-words representation, or BoW, represents each document as a vector with one dimension per vocabulary term. The value in a dimension is usually the number of times that term appears. Word order disappears, but frequency remains.

Take this small corpus:

Document	Text
D1	the cat sat on the mat
D2	the dog sat on the rug
D3	the cat chased the mouse

The raw counts for three terms are:

Document	the	cat	mat
D1	2	1	1
D2	1	0	0
D3	2	1	0

If the query is the, all three documents match, even though the says almost nothing about the subject. A raw dot-product scorer gives D1 and D3 a score of 2 and D2 a score of 1. The repeated function word has become evidence merely because it is common.

That is not a fatal flaw in BoW. Raw counts are useful when repetition itself matters. A spam message containing “free” 12 times may deserve a different score from one containing it once. The problem is that raw counts do not distinguish a term that is frequent because it is important from one that is frequent because English uses it everywhere.

TF-IDF adds that missing distinction.

The two parts of TF-IDF

Term frequency, or TF, measures how prominent a term is within one document. A simple version is:

TF(t, d) = count(t, d) / total_terms(d)

In D1 there are six tokens. Therefore:

• TF(the, D1) = 2 / 6 = 0.333
• TF(cat, D1) = 1 / 6 = 0.167
• TF(mat, D1) = 1 / 6 = 0.167

TF answers: “Is this term important inside this document?”

Inverse document frequency, or IDF, measures how unusual a term is across the corpus. A common unsmoothed formula is:

IDF(t) = log(N / df(t))

Here, N is the number of documents and df(t) is the number of documents containing the term. df counts documents, not occurrences. The two appearances of the in D1 still contribute only one document to its document frequency.

For our corpus, N = 3:

Term	Documents containing it	df	IDF, using natural log
the	D1, D2, D3	3	log(3 / 3) = 0
cat	D1, D3	2	log(3 / 2) = 0.405
mat	D1	1	log(3 / 1) = 1.099

Finally:

TF-IDF(t, d) = TF(t, d) × IDF(t)

For D1:

Term	TF	IDF	TF-IDF
the	0.333	0	0
cat	0.167	0.405	0.068
mat	0.167	1.099	0.183

mat appears only once, but it receives the highest weight because it identifies D1. the appears twice, but receives zero under this particular formula because it appears in every document.

That is the central idea: TF measures local importance; IDF measures global distinctiveness. Their product rewards terms that are both.

A checkable Python implementation

Scikit-learn’s TfidfVectorizer implements this calculation while handling tokenization, vocabulary construction, sparse matrices, IDF values, and normalization.

from sklearn.feature_extraction.text import TfidfVectorizer

docs = [
    "the cat sat on the mat",
    "the dog sat on the rug",
    "the cat chased the mouse",
]

vectorizer = TfidfVectorizer(norm=None, smooth_idf=True)
X = vectorizer.fit_transform(docs)

terms = ["the", "cat", "mat"]
print({
    term: round(vectorizer.idf_[vectorizer.vocabulary_[term]], 3)
    for term in terms
})

The output is:

{'the': 1.0, 'cat': 1.288, 'mat': 1.693}

The values differ from the hand calculation because scikit-learn smooths IDF by default. Its smoothed formula is:

log((1 + N) / (1 + df)) + 1

Smoothing prevents awkward zero values and makes the calculation safer for small or changing corpora. With smoothing, the still gets the lowest IDF, but not zero.

The example also sets norm=None so the IDF values are easy to inspect. In the default configuration, TfidfVectorizer uses raw term counts for TF and then applies L2 normalization to each document vector. L2 normalization divides a vector by the square root of the sum of its squared values. This reduces the advantage of long documents.

After L2 normalization, the dot product between two vectors is their cosine similarity. That is why TF-IDF is commonly used for document retrieval: two documents receive a high similarity when they share distinctive terms, not merely because both contain many words.

What the production pattern looks like

A real pipeline usually follows this sequence:

1. Fit a vectorizer on the training documents or search index. This learns the vocabulary and each term’s IDF.
2. Transform documents into a sparse matrix. A sparse matrix stores only nonzero entries. If the vocabulary has 1 million terms but a document contains 100 distinct terms, that document needs roughly 100 stored values rather than 1 million.
3. Transform queries or new examples with the same fitted vectorizer.
4. Compare vectors for retrieval, or pass them to a classifier such as logistic regression or a linear support vector machine.

The word same matters. At serving time, call transform, not fit_transform, on a query. Fitting again can create a different vocabulary and different IDF values. A classifier trained with 18,432 columns cannot accept a newly fitted query representation with 18,100 columns.

For supervised learning, fit the vectorizer only on the training split. If IDF is calculated using the validation or test documents, information from the evaluation set has entered preprocessing. The labels have not leaked directly, but the evaluation distribution has influenced the features, which makes the measured score less trustworthy. Persist the fitted vectorizer alongside the model.

The senior-level nuance

TF-IDF does not literally remove stop words. It downweights terms that are common in the fitted corpus. With smoothing, a word appearing in every document can still have an IDF of 1. Some common words are also useful. In sentiment analysis, removing not can turn “not good” into “good”, which is a spectacularly efficient way to be wrong.

TF-IDF can also lose to raw counts. If absolute repetition is predictive, normalizing document vectors may erase useful information. Multinomial Naive Bayes is designed around count-like features and often works well with raw counts or closely related weighting. The right choice depends on the task, document length, model, and evaluation metric.

It is also not a semantic representation. “Car” and “automobile” occupy different dimensions. “Dog bites man” and “man bites dog” have the same unigram TF-IDF vector because they contain the same words. A rare typo can receive a very high IDF simply because it occurs once, even though it is noise. Word and character n-grams can add phrase and spelling information, but they do not turn TF-IDF into a language-understanding system.

For search, TF-IDF is an excellent baseline but not always the final ranking method. BM25 usually handles term-frequency saturation and document length more carefully: mentioning a term twice may help, but mentioning it 50 times should not make a document 25 times more relevant. Dense embeddings handle synonyms and related meanings better, but they cost more, are harder to interpret, and can blur exact identifiers such as product codes or error messages. Many strong systems use lexical retrieval, such as BM25 or TF-IDF, alongside embeddings.

A common operational failure appears after a corpus changes. Search results suddenly favor new product IDs, timestamps, or one-off error strings. The first symptom is often a strange top result rather than a crashed service. The cause is usually stale or unstable IDF: the index was rebuilt with a different document population, or the vectorizer was refitted independently for different batches. Refit on a deliberate schedule, keep vocabulary and IDF versioned, and evaluate retrieval on queries that represent the current traffic.

What they’ll ask next

Does TF-IDF remove stop words?

No. It reduces the weight of terms common across the fitted corpus. Explicit stop-word removal is optional and can hurt when words such as not carry task-specific meaning.

When would raw bag-of-words counts be better?

Use counts when repetition or document volume is meaningful, or when using a count-based model such as multinomial Naive Bayes. Compare both representations with the same held-out evaluation rather than assuming TF-IDF must win.

Why use TF-IDF instead of embeddings?

TF-IDF is fast, sparse, cheap, and easy to inspect. It is strong when exact word matching matters, data is limited, or the vocabulary contains domain-specific terms. Embeddings are better at semantic similarity and synonyms, but usually require more computation and careful evaluation. For serious search, testing a hybrid of lexical and semantic retrieval is often sensible.

Say this in the interview

“TF-IDF improves raw bag-of-words by keeping a term’s importance within its document while downweighting terms that appear across many documents, so distinctive words influence retrieval or classification more than generic ones.”