Text preprocessing

A computer can’t do anything with "The cats were running!" as a string — it needs tokens, ideally normalized so that Cats, cats, and cat aren’t three unrelated things. Classical NLP starts with a preprocessing pipeline that turns raw text into clean units. The steps are simple, but each one throws information away on purpose — and knowing what it discards is what tells you when to use it and when to skip it.

The pipeline

import re

text = "The cats were running quickly and the dogs barked loudly!"

tokens = re.findall(r"[a-z]+", text.lower())          # 1. tokenize + lowercase (normalize)

STOP = {"the", "and", "were", "a", "an", "to", "of"}
content = [t for t in tokens if t not in STOP]          # 2. remove stopwords

def stem(w):                                            # 3. crude suffix stripping
    for suf in ("ing", "ly", "ed", "s"):
        if w.endswith(suf) and len(w) - len(suf) >= 3:
            return w[: -len(suf)]
    return w

stems = [stem(t) for t in content]
print("raw     :", text)
print("tokens  :", tokens)
print("content :", content)
print("stems   :", stems)

raw     : The cats were running quickly and the dogs barked loudly!
tokens  : ['the', 'cats', 'were', 'running', 'quickly', 'and', 'the', 'dogs', 'barked', 'loudly']
content : ['cats', 'running', 'quickly', 'dogs', 'barked', 'loudly']
stems   : ['cat', 'runn', 'quick', 'dog', 'bark', 'loud']

Four stages, each shrinking or normalizing the text:

Tokenization is harder than it looks. Splitting on spaces fails fast: "don't" → don + t?, "New York" is one concept in two tokens, "U.S.A." has internal dots, and Chinese and Japanese have no spaces at all. Real tokenizers use language-aware rules. Stopword removal drops ultra-common function words (the, and, of) that carry little topical signal — useful for search and topic models, but it destroys meaning where those words matter (“to be or not to be” becomes empty).

Stemming vs lemmatization

Both collapse inflected forms to a base, but differently:

• Stemming chops suffixes by rule (the Porter stemmer is the classic). It’s fast but crude — notice the demo turned running into runn, not a real word. Stems just need to match consistently, not be correct.
• Lemmatization maps a word to its true dictionary form (its lemma) using morphology and part-of-speech: running → run, better → good, was → be. It’s correct but slower and needs a lexicon.

Use stemming when speed matters and you only need consistent matching (search indexing); use lemmatization when you need real words (linguistic analysis, readable features).

In one breath

• Classical NLP turns raw strings into clean tokens via a pipeline: tokenize → normalize (lowercase) → remove stopwords → stem or lemmatize.
• Tokenization isn’t just splitting on spaces — contractions, multi-word concepts, punctuation, and space-free languages all break naive splitting.
• Stopword removal drops common function words (good for search/topic models, harmful where those words carry meaning).
• Stemming chops suffixes by rule (fast, crude — running → runn); lemmatization maps to the true dictionary form (running → run, correct but slower).
• The whole pipeline is lossy normalization; neural models skip it (subword tokens on near-raw text) because they want the discarded signal — so preprocess for search/BoW, not for transformers.

Quick check

Next

Once you have clean tokens, the next question is how to turn them into numbers a model can use — the classic answer is bag-of-words & TF-IDF. For the subword scheme modern models use instead, see tokenization & BPE.