GloVe & fastText

word2vec was a breakthrough, but it left two openings. It learns locally, sliding a small window over the text one position at a time, never looking at the corpus as a whole. And it gives every word exactly one vector, so a word it never saw during training has no vector at all. Two successors — both from 2014–2016 — each closed one of these gaps.

GloVe: learn from global counts

GloVe (Global Vectors, Pennington et al., Stanford 2014) starts from the opposite end of word2vec. Instead of local prediction windows, it builds the global word-word co-occurrence matrix — for the whole corpus, how often does each word appear near each other word — and then learns vectors whose dot products match the log of those co-occurrence counts.

The insight that makes it work is that ratios of co-occurrence probabilities carry meaning. Take ice and steam: the word solid co-occurs far more with ice than steam, gas the reverse, while water co-occurs with both and fashion with neither. It’s the ratio P(word | ice) / P(word | steam) — large, small, or ~1 — that reveals how a word relates to the ice/steam distinction. GloVe is designed so vector differences capture exactly these ratios, which is why it recovers the same analogy structure as word2vec while training on aggregate statistics rather than streaming windows.

fastText: words made of subwords

fastText (Bojanowski et al., Facebook 2016) keeps word2vec’s predictive training but changes what a word is. A word is represented as the sum of its character n-gram vectors (plus the whole-word vector). where becomes its 3-grams <wh, whe, her, ere, re> and so on. This single change has an outsized payoff: out-of-vocabulary words get a vector for free, assembled from n-grams the model already knows.

# fastText: a word is the bag of its character n-grams, so an unseen word
# still shares n-grams with known words and gets a sensible vector.
def char_ngrams(word, n=3):
    w = f"<{word}>"
    return {w[i:i+n] for i in range(len(w) - n + 1)}

known  = char_ngrams("running")
oov    = char_ngrams("runner")        # pretend 'runner' was never seen as a whole word
shared = known & oov
print("running n-grams:", sorted(known))
print("runner  n-grams:", sorted(oov))
print(f"shared: {sorted(shared)}  ->  OOV 'runner' inherits meaning from {len(shared)} shared n-grams")

running n-grams: ['<ru', 'ing', 'ng>', 'nin', 'nni', 'run', 'unn']
runner  n-grams: ['<ru', 'er>', 'ner', 'nne', 'run', 'unn']
shared: ['<ru', 'run', 'unn']  ->  OOV 'runner' inherits meaning from 3 shared n-grams

runner was never seen whole, yet it shares <ru, run, and unn with running — so fastText composes a meaningful vector for it from pieces it knows. word2vec would have nothing to offer. This also makes fastText strong for morphologically rich languages (Turkish, Finnish) and rare words, where endings and roots recur even when full words don’t.

Three static embeddings, compared

In practice word2vec and GloVe land in a similar place — predictive-local and count-global turn out to be two routes to the same kind of space. All three are static (one vector per word, context-free), the limitation contextual models lift.

In one breath

• word2vec learns locally and gives one vector per word with no fallback for unseen words; GloVe and fastText each fix one gap.
• GloVe factorizes the global co-occurrence matrix (dot product ≈ log co-occurrence count); it works because ratios of co-occurrence probabilities encode meaning (ice vs steam), recovering word2vec’s analogy structure from aggregate statistics.
• fastText represents a word as the sum of its character n-gram vectors, so out-of-vocabulary words get a vector from shared n-grams (the demo: runner inherits from running) — great for rare words and morphologically rich languages.
• All three are static (context-free, one vector per word) — the limit contextual/transformer embeddings remove.
• fastText’s subword idea is the direct ancestor of the subword tokenization in every modern LLM.

Quick check

Next

These finish the static-embedding family; the contextual successors that give a word a different vector per sentence are BERT, GPT, T5. fastText’s subword idea continues in tokenization & BPE, and for using embeddings geometrically see embeddings as vectors.