Why do dense word embeddings outperform one-hot vectors?

How to think about it

Short answer: Dense embeddings usually outperform one-hot vectors because they give the model a learned notion of similarity. “Cat” and “kitten” can have nearby representations, so evidence learned from one can help with the other; one-hot vectors represent only separate identities. The qualification matters: dense is not automatically better, especially when exact identity matters or the embedding was trained on the wrong data.

Why one-hot vectors throw information away

Imagine a support-ticket classifier that must decide whether a message is about pets, billing, or delivery. Its vocabulary contains 50,000 recognized words.

A vocabulary is simply the set of tokens the model knows. With one-hot encoding, cat might be represented as a vector with 50,000 entries:

[0, 0, 0, ..., 1, ..., 0]

The 1 marks the position assigned to cat. kitten gets a different position and a different vector.

This representation records identity perfectly. It does not record meaning.

For any two different one-hot vectors, the dot product is 0, the cosine similarity is 0, and the Euclidean distance is sqrt(2). “Cat” is therefore just as unrelated to “kitten” as it is to “galaxy”. The representation has no way to express partial similarity.

That forces the next model to learn every relationship from labeled examples. If a linear classifier learns that cat is associated with the pet label, it gets no such information for kitten. The classifier has one parameter for one word and a separate parameter for the other.

One-hot vectors are also wide. A 50,000-entry vector has 50,000 coordinates, although only one is nonzero. A sensible implementation does not materialize all those zeros; it stores the token ID and performs a lookup. So the problem is not that every request necessarily moves 50,000 floating-point numbers through memory. The deeper problem is the lack of shared structure.

What a dense embedding changes

A dense embedding is a relatively small vector of real numbers learned so that useful relationships are reflected in its geometry. Dense means that many coordinates carry values rather than being almost entirely zero.

Instead of 50,000 coordinates, a classic word embedding might use 100, 200, or 300 dimensions. Modern language models often use vectors with hundreds or thousands of dimensions.

The model learns these vectors from context. This relies on the distributional hypothesis, the idea that words appearing in similar surroundings tend to have related uses. “Cat” and “kitten” may both appear near words such as “pet”, “feed”, and “sleep”. Over many examples, their vectors tend to move toward similar regions.

A typical embedding table is a matrix E with V rows and d columns:

E ∈ R^(V × d)

Here, V is the vocabulary size and d is the embedding dimension. The vector for token i is row i of E.

Word2Vec learns vectors by predicting nearby words. GloVe learns from aggregate co-occurrence statistics. fastText adds character n-gram information, which helps with rare and previously unseen word forms. The algorithms differ, but the useful outcome is the same: words with related distributional behavior receive related vectors.

The information is distributed, meaning no single coordinate means “animal” or “plural”. Several coordinates together may encode many overlapping properties. That makes the representation useful for generalization: the model can respond to a pattern rather than memorizing an isolated address.

There is an important implementation detail here. If a one-hot row vector x is multiplied by a weight matrix W, the result xW is just the row of W selected by the position of the 1. In other words, a one-hot input followed by a linear layer is mathematically an embedding lookup. Dense embeddings do not win merely because lookup is faster.

The advantage comes from what happens after the lookup and from how the table was learned. A 100-dimensional vector gives later layers a shared space in which related words can influence similar computations.

A concrete comparison

For a 50,000-word vocabulary and a 100-dimensional embedding, the embedding table contains 5,000,000 values. At four bytes per 32-bit floating-point value, that is about 20,000,000 bytes, or 19.1 MiB.

A fully materialized one-hot vector would contain 50,000 floating-point values, about 195 KiB for one token. But production code normally stores only its integer ID. The parameter comparison is more revealing:

• A layer accepting 50,000 input values and producing 128 hidden values needs 6,400,000 weights.
• A layer accepting 100-dimensional embeddings and producing 128 hidden values needs only 12,800 weights.
• The 50,000-by-100 embedding table still contains 5,000,000 weights if it is learned.

So the embedding does not make all parameters vanish. It moves the model toward a compact interface for the rest of the network and, more importantly, gives that interface useful geometry.

Here is a deliberately tiny example. These vectors are hand-written to make the geometry visible; a real model would learn them.

import numpy as np

vectors = {
    "cat": np.array([0.8, 0.6, 0.0]),
    "kitten": np.array([0.7, 0.7, 0.1]),
    "dog": np.array([0.2, 0.8, 0.2]),
    "galaxy": np.array([0.0, 0.0, 1.0]),
}

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

for left, right in [
    ("cat", "kitten"),
    ("cat", "dog"),
    ("cat", "galaxy"),
]:
    print(f"{left} vs {right}: {cosine(vectors[left], vectors[right]):.3f}")

The output is:

cat vs kitten: 0.985
cat vs dog: 0.754
cat vs galaxy: 0.000

The high cosine similarity between cat and kitten gives a downstream model a chance to reuse evidence. A classifier trained on 10,000 support tickets might see “cat” 600 times but “kitten” only 40 times. With one-hot features, those words have separate parameters. With a suitable pretrained embedding, the classifier can learn a useful pet-related direction from the richer evidence around both words.

That is statistical sharing. It is the central reason embeddings tend to need fewer labeled examples.

The nuance that earns the senior signal

Do not say that dense embeddings always encode true meaning. They encode patterns in their training data.

“Hot” and “cold” may be close because they appear in similar sentences, even though they are opposites. “Nurse” may be closer to gendered terms because of stereotypes in the corpus. A high cosine similarity means similar usage, not synonymy, approval, causation, or fairness.

Warning — similarity is not truth. Embeddings can reproduce social bias, domain bias, and accidental correlations. Audit nearest neighbors and task-level behavior before shipping a pretrained embedding, particularly in hiring, lending, health, or moderation systems.

There is also a difference between static and contextual embeddings. A Word2Vec-style model gives “bank” one vector whether the sentence concerns a riverbank or a financial institution. A transformer produces a representation that depends on the surrounding sentence, so the two uses can separate. When an interviewer says “word embeddings”, clarify which kind they mean.

The real advantage may also come from pretraining rather than density. A randomly initialized 100-dimensional embedding trained on 200 labeled examples has no reason to place “cat” near “kitten”. It may perform worse than a one-hot baseline. A pretrained embedding has already absorbed information from a much larger corpus, which is why it can transfer useful structure to a small task.

One-hot features remain a good choice when exact identity is the point:

• A small vocabulary makes the width harmless.
• A product SKU, user ID, or error code may have no meaningful notion of “nearby”.
• A symbolic system may need to distinguish every token exactly.
• One-hot or hashed features can be easier to inspect and debug.
• A large, well-labeled dataset may let a one-hot model learn task-specific weights effectively.

Embedding dimension is a trade-off, too. A vector that is too small may collapse distinctions and underfit. A very large vector costs memory, increases serving and training work, and can overfit when labeled data is scarce. I would compare against a one-hot baseline, tune the dimension on held-out data, and inspect both task metrics and nearest neighbors.

A production failure to watch for

A common failure appears when a word-level embedding meets a new domain. Suppose an embedding trained on news articles is used for customer-support tickets. The offline validation score looks acceptable, but production recall drops for new product names, abbreviations, and internal error codes.

The first symptom may be that many unfamiliar tokens become the same <UNK> representation, so unrelated tickets look artificially similar. The cause is out-of-vocabulary coverage and domain mismatch, not necessarily a bad classifier. Subword methods such as fastText, the tokenizer used by a modern language model, domain-specific pretraining, or a carefully chosen unknown-token strategy can help. Always measure performance separately on common words, rare words, and unseen forms.

What they’ll ask next

“Is a dense embedding always cheaper than one-hot?”
No. A one-hot vector is usually stored as an ID, and one-hot plus a linear projection is the same operation as an embedding lookup. Dense embeddings reduce the width seen by later layers and provide learned structure, but the embedding table still has parameters and memory cost.

“Do embeddings understand semantics?”
Not in the human sense. They capture statistical relationships in their training data. They can encode useful similarity, but they can also confuse related context with synonymy and inherit bias. Contextual embeddings handle multiple meanings better than one fixed vector per word.

“How would you handle unseen words?”
I would measure the out-of-vocabulary rate first, then consider subword features, a tokenizer that decomposes unfamiliar words, domain adaptation, or a fallback representation. I would also test rare and new terms separately instead of trusting an aggregate validation score.

Say this in the interview

“Dense embeddings usually beat one-hot vectors because they replace isolated word IDs with a learned low-dimensional geometry, allowing models to share information between related words; but the gain comes from the training signal and domain fit, not from density alone, so I would still keep a one-hot baseline for exact-identity or small-vocabulary tasks.”