What are stop words and when should you remove them?

How to think about it

Stop words are very common function words, meaning words whose main job is grammar rather than naming a topic, such as “the”, “is”, “and”, or “to”. Remove them selectively in count-based or TF-IDF pipelines when they add noise; keep them when negation, question form, phrase meaning, sequence, or a pretrained model depends on them. There is no universally correct stop-word list.

Why remove them at all?

An interviewer is usually testing whether you understand the difference between “frequent” and “useless”. They are not the same thing.

In a bag-of-words representation, a document becomes a vector of token counts. A token is a piece of text produced by tokenization, such as a word or punctuation mark. If your vocabulary contains 50,000 tokens, every document is represented using 50,000 possible features, even though a particular document may contain only 100 of them. That is a sparse vector: a long vector containing mostly zeros.

Words such as “the” and “is” often appear in almost every document. They therefore tell a classifier or search engine little about the document’s subject. If every support ticket contains “the”, the feature cannot help much with distinguishing a battery problem from a billing problem.

TF-IDF, or term frequency–inverse document frequency, is a weighting scheme that already reduces this effect. Its inverse-document-frequency part is commonly written as:

idf(word) = log(N / df(word))

Here, N is the number of documents and df(word) is the number of documents containing the word. In a corpus of 10,000 documents, a word appearing in all 10,000 has an unsmoothed IDF of:

log(10,000 / 10,000) = 0

A word appearing in only 100 documents has an IDF of:

log(10,000 / 100) = log(100) ≈ 4.605

So ubiquitous words naturally receive little weight in TF-IDF. Libraries often use a smoothed version, so the exact value may not be zero, but the principle remains.

Explicitly removing stop words can still help. It reduces the number of stored tokens, vocabulary columns, inverted-index postings, and sometimes model-training work. The gain depends on the corpus. If your vocabulary has 50,000 terms and your stop list removes 180, you have removed only 0.36 percent of the possible columns. That may be irrelevant for a modern classifier. If those 180 words occur millions of times in a search index, however, removing their postings may still reduce storage and query work.

That is why “stop-word removal reduces dimensionality” is technically true but not automatically a meaningful performance improvement.

A concrete example: support tickets

Suppose you are building search for 10,000 device-support tickets. Two tickets contain:

Ticket	Original text	After removing the, is, and not
A	The device is not charging	device charging
B	The device is charging	device charging

The two tickets become identical. A search for “not charging” can now retrieve the same representation as “charging”. A classifier cannot distinguish a broken device from a working one.

If not is retained, the binary presence vectors over [device, not, charging] are:

A = [1, 1, 1]
B = [1, 0, 1]

Their cosine similarity, a measure of how closely two vectors point in the same direction, is approximately:

2 / (sqrt(3) * sqrt(2)) ≈ 0.816

If not is removed, both vectors are [1, 1], and their cosine similarity becomes exactly 1.0. The system has lost the distinction entirely.

A small implementation makes the policy visible:

text = "The device is not charging"
stop = {"the", "is"}  # Keep "not" for this task

tokens = text.lower().split()
kept = [token for token in tokens if token not in stop]

print(kept)
# ['device', 'not', 'charging']

This is deliberately simple. A production tokenizer must also handle punctuation, contractions, Unicode, and language-specific rules. The important detail is that filtering happens on complete tokens. A naive substring replacement could remove the from theater, which would be an impressive way to damage a vocabulary.

When removal is a reasonable choice

For classical sparse models, stop-word removal is worth testing. This includes TF-IDF with logistic regression, linear support-vector machines, or Naive Bayes; topic classification; and some keyword-based retrieval systems. Compare a baseline with no removal against a version using a carefully reviewed list. Keep the version that wins on the task metric, not the version that sounds cleaner.

For ordinary topical classification, words such as “the” and “is” often contribute little. Removing them can make the feature matrix smaller and sometimes improves generalization by eliminating weak, noisy features. But regularization can already give those features very small weights, so the accuracy gain may be zero.

Search requires more care. Removing common words from both documents and queries can reduce index size and make keyword matching faster. It can also break exact phrase search, legal search, quoted queries, and short queries where every word matters. A user searching for “to be or not to be” is not asking for the same thing as a user searching for “be”.

For sentiment analysis, intent classification, question answering, and named entity recognition, keep function words by default. “Not good” and “good” have opposite sentiment. “Who approved the refund?” and “What approved the refund?” have different question types. In named entity recognition, surrounding grammar can help the model decide where an entity starts and ends.

For language modeling, keep them. The model must learn that “the cat sat” and “cat the sat” have different grammatical structure.

For fine-tuning a pretrained transformer, normally keep the original text and let the model’s tokenizer handle it. A transformer is a neural architecture that uses token representations and their positions to model context. Removing words changes the sequence, shifts positions, and can remove exactly the relationships the model learned during pretraining. This is not the same as removing a low-value column from a traditional sparse matrix.

The senior-level nuance

A stop word is task-dependent. It is not simply a short word, a function word, or a word appearing frequently.

The word “who” may be low-value for topic classification and essential for question answering. The word “will” may be grammatical in one sentence and part of a person’s name in another. “Against” might be noise in a broad news classifier but crucial in a legal or financial search system.

Generic lists also differ. One library may include not, no, or never; another may omit one of them. Some lists are designed for information retrieval, others for linguistic preprocessing. Treating a library’s default list as a law of nature is a common beginner mistake.

A domain corpus may need its own policy. In a legal collection, court, case, and law may appear in nearly every document. They could be useful stop-word candidates for a narrow topic classifier. They should probably remain searchable in a legal search engine because a query for “court case” is meaningful.

A sensible custom list starts with document frequency, the number of documents containing each token, but does not end there. For a classifier, check whether a candidate has an association with the target label. For retrieval, evaluate judged queries using measures such as Recall at k, which asks whether a relevant result appears among the top k results. Use the training or reference corpus to design the list, then test it on untouched evaluation data.

Apply the same preprocessing everywhere. If training removes not but the serving path keeps it, the model sees a different feature space at inference time. For retrieval, normalize the indexed documents and incoming queries consistently. Keep the original raw text for debugging and auditability, and version the stop-word list like any other production configuration.

Do not silently combine stop-word removal with stemming or lemmatization. Stemming reduces words to crude roots, while lemmatization maps them to dictionary forms. Each choice can change features, so changing all three at once makes it difficult to know why a model improved or failed.

A failure mode you can recognize

The first symptom of harmful stop-word removal in the support-ticket system is usually behavioral: the query not charging returns nearly the same top results as charging, or the classifier starts labeling “not eligible” as “eligible”. A confusion matrix then shows more false positives, especially around negated phrases.

The root cause is often a generic list that removed not, or a preprocessing mismatch between training and serving. The fix is not merely “add more stop words”. Restore critical tokens, add tests for negation and phrase queries, and compare the result against a no-removal baseline.

What they’ll ask next

Doesn’t TF-IDF already solve the stop-word problem?

Often, partially. IDF downweights words that occur in many documents, so removing them may produce little accuracy improvement for a TF-IDF model. Explicit removal can still reduce feature storage or search-index postings. It is most useful when the operational savings are measurable or validation shows a task benefit.

Should I always keep not?

Keep it whenever negation can change the label or meaning, but protecting not alone is not a complete negation system. Phrases such as “never works”, “hardly useful”, and “no longer active” also matter. For sentiment, consider preserving negation scope or adding phrase features such as not_good, then validate on examples where the polarity changes.

Would you remove stop words before BERT or another transformer?

Normally, no. Keep the original text and use the model’s normal tokenizer. Generic stop-word deletion can remove semantic information and alter token positions. The exception is a pipeline deliberately trained and evaluated with that preprocessing; the decision should follow evidence from that exact model and task.

Say this in the interview

“Stop-word removal is a task-dependent optimization: I would test it for sparse TF-IDF or retrieval features, but keep negation, question words, phrase context, and the original input for tasks or models that depend on them.”