What is the difference between stemming and lemmatization?

How to think about it

The direct answer

Stemming reduces a word to a rough character stem by applying suffix-stripping rules, so it is fast but may produce a non-word such as studi. Lemmatization reduces a word to its dictionary form, called a lemma, using vocabulary and morphological information, often including the word’s grammatical role.

I would choose based on the task, not on the claim that one is universally better. Stemming is often a useful, cheap choice for keyword search. Lemmatization is more interpretable, but it costs more and depends on good language resources and part-of-speech tags.

Why the distinction matters

Imagine a shoe shop with a search box. A customer searches for run shoes, but a product description says:

These shoes are ideal for running.

A literal text matcher sees run and running as different tokens. The customer may miss a perfectly relevant product because English has changed the word’s grammatical form.

An inflected form is a grammatical variation of a word. For example, run, runs, running, and ran are related forms, although they do not all have the same spelling. Normalization tries to bring related forms together so a search system or classifier does not treat every variation as an unrelated feature.

Stemming and lemmatization solve that problem differently.

How stemming works

A stemmer applies a fixed sequence of rules to the characters in a word. The classic Porter stemmer and the later Snowball stemmers are examples. They do not need to understand the sentence or consult a dictionary. They ask questions such as whether a suffix is present and whether the remaining string is long enough to justify removing it.

That makes stemming fast, deterministic, and easy to run at indexing time. It also makes it blunt.

For example, an English stemmer might produce:

Word	Stem
running	run
studies	studi
generous	gener

The last two results are not intended to be English words. They are internal matching keys. The stemmer’s goal is to make related strings collide, not to produce something a person would put in a dictionary.

That shortcut can cause over-stemming, where unrelated words are pushed into the same bucket. A Porter-style stemmer can map both generous and generate to gener. In a search index, that may create an unwanted match. It can also cause under-stemming, where related forms remain separate because the rules do not recognize their relationship.

Stemming is therefore not truly language-agnostic. The algorithm may operate only on characters, but its rules are language-specific. An English Porter stemmer is not an appropriate Spanish or German stemmer. Other languages need their own rules, and some languages have much richer morphology than English.

How lemmatization works

A lemmatizer tries to find the word’s lemma, meaning its canonical dictionary form. It uses a lexicon and morphological rules, and it often needs a part-of-speech tag, which identifies the word’s grammatical role as a noun, verb, adjective, or another category.

That grammatical role is not a detail. It can change the answer:

• running, used as a verb, can become run
• studies, used as a verb or plural noun, can become study
• better, used as an adjective, can become good
• saw, used as a noun, stays saw; used as a verb, it can become see

A lemmatizer is not merely a more sophisticated suffix stripper. It can use irregular forms stored in its vocabulary. No simple rule that removes the ending from better would reliably discover good.

Here is a small NLTK example. The v and a values mean verb and adjective.

from nltk.stem import PorterStemmer, WordNetLemmatizer

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

examples = [
    ("running", "v"),
    ("studies", "v"),
    ("better", "a"),
]

for word, pos in examples:
    print(word, stemmer.stem(word), lemmatizer.lemmatize(word, pos=pos))

With the WordNet corpus available, it prints:

running run run
studies studi study
better better good

The same lemmatizer can return a different result if you give it the wrong grammatical role. That is why a production lemmatization pipeline commonly includes part-of-speech tagging before lemmatization.

&l t;Callout type=“warn”> Lemmatization does not guarantee perfect meaning or even a useful output for every token. An unknown product name may be left unchanged, and a bad part-of-speech tag can produce the wrong lemma. Lemmatization handles morphology; it is not a thesaurus. &l t;/Callout>

A concrete search example

Consider this tiny shoe catalogue:

Document	Text
D1	I run in lightweight shoes.
D2	I was running in lightweight shoes.
D3	I was running a marathon.

The query is run shoes.

Suppose a toy search engine requires both query terms to appear exactly. Without normalization, D1 matches both run and shoes. D2 contains shoes, but it contains running, not the exact token run. D3 does not contain shoes.

Now normalize both the documents and the query:

• run becomes run
• running becomes run
• shoes becomes shoe under a plural-handling rule

D1 and D2 now match. If D1 and D2 are the relevant products, recall rises from one relevant result out of two to two out of two.

Recall is the fraction of all relevant results that the system returns. Precision is the fraction of returned results that are actually relevant. Normalization often improves recall because it joins variants. It can reduce precision if it also joins words that should remain distinct.

The production detail that matters is consistency: the same analyzer must be applied to both sides. If documents are stemmed when they are indexed but the user’s query is left untouched, the index may contain run while the query asks for running. The first symptom is variant-dependent search behavior: run shoes works, while running shoes unexpectedly returns fewer or no results.

A search system can also preserve both forms. It might store the original text for exact phrase matching and a normalized field for broader matching. That gives the ranking system more options than forcing every query through one irreversible transformation.

The senior-level nuance

The textbook answer often says, “Use lemmatization when accuracy matters.” That is incomplete.

Lemmatization is not automatically more accurate for every downstream task. It may improve a small word-count classifier by reducing sparsity, but it can also erase useful information. The difference between ship and shipping, or between singular and plural nouns, may matter to a classifier. A lemmatizer can remove that signal.

For a conventional keyword search system, stemming is attractive because it is cheap and predictable. The index may contain millions of tokens, and applying a few string rules is usually much less expensive than running a full linguistic pipeline. But the right stemmer, language, ranking behavior, and collision rate still need evaluation.

For a bag-of-words classifier, meaning a model that represents a document largely by word counts, I would test three versions:

1. Original tokens
2. Stemmed tokens
3. Lemmatized tokens

I would compare them on a fixed validation set rather than assuming lemmatization wins. A reduction that looks linguistically elegant can perform worse if the task depends on tense, plurality, or domain-specific terminology.

For a pretrained transformer, embedding model, named-entity recognizer, or question-answering system, I would usually preserve the original text unless the model or task specifically calls for normalization. These models were trained on natural word forms and use subword tokenization. Turning running into run before tokenization changes the input distribution and may remove useful context. A model that understands morphology may already handle the relationship internally.

Here is an illustrative search trade-off, not a universal benchmark:

• Without normalization, a query returns 6 of 10 relevant documents and no irrelevant ones. Recall is 60 percent and precision is 100 percent.
• With aggressive stemming, it returns 9 relevant documents and 4 irrelevant ones. Recall is 90 percent, but precision is about 69 percent.

The second system is better if missing a relevant product is expensive. The first may be better if users strongly dislike unrelated results. The business cost decides which error matters.

The other practical issue is language coverage. WordNet lemmatization is useful for English, but a lemmatizer for another language needs an appropriate lexicon, morphological analyzer, and tagger. A resource-poor language may get worse results from a supposedly sophisticated lemmatizer than from a carefully tested stemmer.

What they’ll ask next

Is lemmatization always better than stemming?

No. Lemmatization usually produces more linguistically meaningful forms, but it is slower, resource-dependent, and sensitive to part-of-speech errors. Stemming can be the better engineering choice for a large lexical index when fast, broad matching matters more than readable output.

Why does part-of-speech tagging matter?

Because the same spelling can have different lemmas in different grammatical roles. Better as an adjective can map to good, while better as a verb generally remains better. Saw as a noun is saw; as a verb, it can map to see. A lemmatizer without the right tag may return a valid-looking but incorrect form.

Should I stem or lemmatize text before sending it to a transformer or an embedding model?

Usually neither. Keep the natural text unless experiments show a clear benefit or the model’s documentation requires preprocessing. For search, use normalization in the search analyzer. For semantic embeddings and modern language models, let the model’s tokenizer and learned representations handle word variation, then verify the choice with task-specific evaluation.

Say this in the interview

“Stemming uses fast heuristic rules and may create non-words, while lemmatization uses vocabulary, morphology, and often part of speech to find a canonical form; I choose between them based on the precision–recall trade-off and the downstream model, rather than assuming lemmatization is always better.”