word2vec: CBOW & skip-gram

Bag-of-words and TF-IDF have a fatal blind spot: every word is its own isolated dimension, so car and automobile are as unrelated as car and banana. The representation has no notion of meaning. word2vec (Mikolov et al., 2013) fixed this and launched modern NLP: it learns a dense vector per word — a few hundred numbers — such that words used in similar ways end up close together, learned from nothing but raw text.

Meaning from context

The principle is the distributional hypothesis: “you shall know a word by the company it keeps.” Words appearing in similar contexts have similar meanings. word2vec turns this into a prediction task and trains a shallow neural network on it — and the weights it learns become the embeddings. There are two symmetric formulations:

• CBOW (continuous bag-of-words) — predict the center word from the surrounding context. Faster, and slightly better for frequent words.
• Skip-gram — predict the surrounding context from the center word. Slower but stronger on rare words and small corpora; the more commonly used of the two.

We never actually care about the prediction. We train the network, then throw away the output layer and keep the hidden weight matrix — one row per word. Those rows are the embeddings.

The payoff: relationships become directions

Because words are placed by how they’re used, the geometry of the space encodes relationships as directions — and you can do arithmetic on meaning. The famous example:

import numpy as np

# Toy 2-D vectors hand-placed so x = gender, y = royalty (shows the mechanism).
vec = {
    "king":  np.array([ 1.0,  1.0]),
    "queen": np.array([-1.0,  1.0]),
    "man":   np.array([ 1.0, -1.0]),
    "woman": np.array([-1.0, -1.0]),
}
result = vec["king"] - vec["man"] + vec["woman"]      # king - man + woman
def cos(a, b): return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
best = max((w for w in vec if w not in {"king", "man", "woman"}), key=lambda w: cos(result, vec[w]))
print("king - man + woman =", result, " -> nearest word:", best)

king - man + woman = [-1.  1.]  -> nearest word: queen

Subtracting man and adding woman moves you along the gender direction, landing on queen. Real word2vec discovers these directions on its own from text — gender, plurality, verb tense, even country–capital — without ever being told they exist. (Training stays cheap via negative sampling: instead of a full softmax over the whole vocabulary each step, the model just learns to tell a true context word from a few random “negative” ones.)

The catch: one vector per word, forever

word2vec embeddings are static: each word gets exactly one vector, no matter the sentence. So bank (river) and bank (money) collapse into a single compromise point, and the model can’t tell which you meant. That limitation is precisely what contextual embeddings (BERT/GPT and the transformer) solve — they produce a different vector for a word depending on its sentence.

In one breath

• word2vec learns a dense vector per word from raw text, so similar words sit close — fixing bag-of-words’ “no meaning” blind spot.
• It’s built on the distributional hypothesis (“a word is known by the company it keeps”) via a prediction task: CBOW (context → word) or skip-gram (word → context); the hidden weights become the embeddings.
• The payoff is analogy arithmetic — king - man + woman ≈ queen — because relationships become directions the model discovers unsupervised (trained cheaply with negative sampling).
• Its limit is being static: one vector per word, so bank (river/money) can’t be distinguished — what contextual embeddings (BERT/transformers) fix.
• The big idea — meaning learned from unlabeled text as a vector — runs straight to GloVe/fastText and every modern LLM.

Quick check

Next

word2vec is predictive and per-word; GloVe and fastText refine it with global statistics and subwords. For what these vectors are geometrically and how to use them, see embeddings as vectors; for the contextual successors, BERT, GPT, T5.