What are n-grams and when should you use them in NLP?

How to think about it

An n-gram is a contiguous sequence of n tokens, such as words or subwords. Use n-grams when short-range word order carries useful signal and a fast, interpretable model is a good fit; do not treat them as a general replacement for dense learned vectors, called embeddings, that capture broader semantic similarity.

Why n-grams work

A bag-of-words representation records which tokens appear in a document, and often how many times, but throws away their order. The sentences "dog bites man" and "man bites dog" therefore have the same unigram features even though they describe different events.

An n-gram puts some of that order back.

• A unigram contains one token: "charge".
• A bigram contains two adjacent tokens: "charge reversed".
• A trigram contains three: "card charge reversed".

This matters because a classifier can assign a separate weight to "not reversed" rather than treating "not" and "reversed" as unrelated pieces. The model now has a feature representing a local interaction.

The word local is doing important work. A bigram only sees adjacent tokens. It does not connect "not" to "approved" in "not immediately approved"; that relationship requires other features, such as trigrams, or a model that handles longer context.

N-grams are usually converted into columns in a feature matrix. Each row is a document. Each column is a unigram or phrase. The value might be a raw count, a binary indicator, or a TF-IDF score. TF-IDF means term frequency multiplied by inverse document frequency: it gives more weight to terms that matter in a document but appear in fewer documents overall.

A logistic regression or linear support-vector classifier can then learn a weighted sum of these columns. The model is simple, but the feature engineering gives it useful phrase-level information.

A concrete example

Suppose we are routing customer-support tickets. Some concern account access, and others concern card payments:

from sklearn.feature_extraction.text import CountVectorizer

tickets = [
    "password reset link expired",
    "password reset link works",
    "card charge reversed",
    "card charge not reversed",
]

vectorizer = CountVectorizer(ngram_range=(1, 2))
X = vectorizer.fit_transform(tickets)

print(X.shape)
print("reset link" in vectorizer.get_feature_names_out())
print(X[:, vectorizer.vocabulary_["reset link"]].toarray().ravel())

The output is:

(4, 17)
True
[1 1 0 0]

There are four ticket rows and 17 observed features: nine unigrams and eight bigrams. The "reset link" column is active for the first two tickets.

The bigram "not reversed" is another useful feature. It lets a classifier distinguish "card charge reversed" from "card charge not reversed" using a single feature. A unigram model can use the separate "not" feature, but that word may appear in many unrelated contexts. The bigram is more specific.

In a real classifier, I would normally keep both unigrams and bigrams. The ngram_range=(1, 2) setting means “include orders one and two,” not “use only bigrams.” Unigrams help when a new phrase has never appeared during training. Bigrams add the phrase-level signal when the exact pair is known.

A typical production pattern looks like this:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

model = Pipeline([
    ("features", TfidfVectorizer(
        ngram_range=(1, 2),
        min_df=2,
        max_df=0.98,
        sublinear_tf=True,
    )),
    ("classifier", LogisticRegression(max_iter=1000)),
])

model.fit(train_tickets, y_train)

Here, min_df=2 discards a feature that occurs in only one training document. That reduces noise and memory use. max_df=0.98 discards features appearing in almost every document because they rarely help distinguish classes. The pipeline also ensures that the vectorizer is fitted on the training data rather than accidentally learning its vocabulary from validation or test text.

N-gram features are not the same as n-gram language models

The same term describes two related but different techniques.

An n-gram feature extractor turns text into inputs for a classifier or regressor. Its question is: “Which phrases appear in this document?”

An n-gram language model estimates the next token from the previous n - 1 tokens. Its question is: “Given this context, what token is likely next?”

For a bigram language model, the maximum-likelihood estimate is:

P(next token | previous token) = count(previous token, next token) / count(previous token)

If a corpus contains the token "reset" 100 times and the phrase "reset link" 30 times, the estimated probability of "link" after "reset" is 0.30.

A trigram model instead uses two previous tokens, such as:

P(link | password, reset)

Increasing n gives the model more context, but it also creates more unseen sequences. If "password reset link" never appeared in training, a raw trigram model may assign it probability zero even though it is perfectly sensible. Smoothing, interpolation, or backoff methods reduce this problem by borrowing information from shorter n-grams.

That distinction is worth stating in an interview. N-grams can be useful features without being used to generate text, and a language model has probability and unseen-sequence problems that a classifier feature matrix does not have in quite the same form.

When I would use them

Situation	Good starting point	Why
Small or medium text-classification data	Word unigrams plus bigrams with TF-IDF	Fast, strong baseline with understandable features
Sentiment, intent, spam, or support routing	Word bigrams, sometimes trigrams	Short phrases such as "refund requested" carry signal
Typos, OCR, usernames, product codes, or noisy text	Character n-grams, often three to five characters	Shared character fragments survive spelling variation
Autocomplete or a constrained language model	Smoothed n-gram language model	Cheap, predictable, and easy to run offline
Semantic search or long-distance reasoning	Embeddings or a neural language model	N-grams do not understand synonyms or distant relationships

Character n-grams are particularly useful when the exact word vocabulary is unstable. "colour" and "color" share many character fragments, while a word-level vectorizer treats them as unrelated unless preprocessing connects them. The trade-off is interpretability: "olo" is a less satisfying feature to explain than "card charge".

N-grams are also a good baseline. Before deploying a large neural model for a ticket-routing problem, I would train a word-level TF-IDF model with unigrams and bigrams. It gives a quick reference point, exposes label problems, and often performs surprisingly well when the classes are separated by recurring phrases.

The senior-level nuance

More context is not automatically better.

If the vocabulary contains 50,000 distinct words, there are up to 2.5 billion ordered bigram combinations before considering trigrams. Most will never occur, so the matrix is sparse, meaning most entries are zero. Sparse storage helps, but the vocabulary and the nonzero entries can still become large.

A four-gram may capture a very precise phrase in training and then be useless for the next customer because one word changed. This is the classic precision-versus-generalisation trade-off:

• Unigrams generalise better but lose word order.
• Bigrams capture common local phrases at a manageable cost.
• Trigrams and higher orders capture more detail but need substantially more data.
• Very high orders often memorise instead of generalising.

I would choose n with validation data and an operational memory budget, not by assuming that the largest value wins. min_df, a maximum feature limit, and a sparse linear model are practical controls. Hashing-based vectorization can also cap the explicit vocabulary, though hash collisions make individual features harder to interpret.

Preprocessing changes the answer too. Tokenization determines what counts as a token. Lowercasing makes "New York" and "new york" match, while removing punctuation may erase useful signals in URLs, code, or product identifiers. I would not remove negation words casually. If "not" disappears during preprocessing, the model cannot form the feature "not approved".

Common misconception: a bigram does not “understand” the phrase "not approved". It memorises that those two tokens appeared next to each other and lets the downstream model learn a weight for that pattern. It will not reliably recognise "the request was not approved" as the same relationship unless other features or a more capable model help.

Failure modes to watch for

The first symptom of uncontrolled n-gram growth is often a training process that consumes steadily more memory, slows during vectorization, or is killed by the operating system. The cause is usually an overly large maximum n, character features over unrestricted text, or many one-off tokens. Raise min_df, cap features, restrict the n-gram range, or use a hashing approach.

Another symptom is that production predictions collapse toward one class. Inspect the number of nonzero features in transformed documents. If new tickets contain mostly unseen words, a word-level vectorizer may produce nearly empty rows. A linear classifier then relies mostly on its intercept, which often means predicting the majority class. Character features, better normalization, or retraining on representative text can help.

A suspiciously high validation score followed by a sharp production drop often means leakage or memorisation. Fit the vectorizer only on the training split, and use a time-based split when language changes over time. A random split can place nearly identical templates in both training and validation, making rare n-grams look more useful than they really are.

For n-gram language models, the obvious failure is a zero probability for an unseen phrase. Smoothing and backoff are not optional details; they are what stop one missing count from making an entire sentence impossible.

What they’ll ask next

Are n-grams only used for language modelling?
No. They are also feature representations for classification, ranking, search, spelling correction, and anomaly detection. In those settings, the n-gram is an input feature rather than a next-token probability.

Why combine unigrams and bigrams instead of using bigrams alone?
Unigrams provide coverage when a new phrase has not appeared in training. Bigrams add local order when a familiar phrase is present. The combination usually gives a better balance between generalisation and specificity.

How would you handle unseen n-grams?
For a classifier, the vectorizer ignores features outside its fitted vocabulary, so I would monitor unknown-feature rates and the number of nonzero features. For a language model, I would use smoothing, interpolation with shorter n-grams, or a backoff model. If unseen wording is frequent, character n-grams or a neural representation may be a better fit.

Say this in the interview

“N-grams are contiguous token sequences that restore limited word order to text features; I use word unigrams and bigrams for fast, interpretable classification, character n-grams for noisy text, and avoid high orders when sparsity and poor generalisation outweigh the extra context.”