Multilingual NLP

Almost everything in this course is implicitly about English — but there are roughly 7,000 languages, most with a tiny fraction of English’s labelled data. Multilingual NLP asks: can one model serve many languages, and — crucially — can we transfer (reuse) what we learn from a high-resource language to a low-resource one? The enabling idea is a shared embedding space.

A shared space where translations meet

Train embeddings (lists of numbers that represent each word) so that words meaning the same thing in different languages land in the same region of the vector space. Then “dog,” “chien,” and “perro” are neighbors — and a model that learned to do something with the English vectors works on the others, because they’re in the same place:

import numpy as np

emb = {"dog": [1.0, 0.1], "chien": [0.9, 0.2], "perro": [1.0, 0.0],     # 'dog' across languages
       "cat": [0.1, 1.0], "chat":  [0.2, 0.9], "gato":  [0.0, 1.0]}     # 'cat' across languages

def cos(a, b):
    a, b = np.array(a), np.array(b)
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

q = "dog"
for w in sorted((w for w in emb if w != q), key=lambda w: cos(emb[q], emb[w]), reverse=True)[:3]:
    print(f"nearest to 'dog': {w:<6} cos={cos(emb[q], emb[w]):.2f}")

nearest to 'dog': perro  cos=1.00
nearest to 'dog': chien  cos=0.99
nearest to 'dog': chat   cos=0.31

The nearest neighbors of dog are its translations perro and chien — not the (English) word cat. That alignment is the whole trick: meaning, not language, decides position.

Cross-lingual transfer

This is what makes zero-shot cross-lingual transfer possible. Fine-tune a multilingual model on an English task with English labels (say, named-entity recognition), then run it on Swahili or Hindi text it was never trained on — and it often works, because the multilingual encoder (the part of the model that turns text into vectors) maps both languages into the shared space. The workhorses are mBERT (multilingual BERT, pretrained on 104 languages’ Wikipedia) and XLM-R (trained on 100 languages of web text), which learn this aligned space from raw multilingual text without parallel translations.

In one breath

• Multilingual NLP serves many of the world’s ~7,000 languages and aims to transfer from high-resource (English) to low-resource ones.
• The key idea is a shared embedding space where translations cluster (dog/chien/perro are neighbors) — position encodes meaning, not language.
• That enables zero-shot cross-lingual transfer: fine-tune on an English task, then apply to a language with no labelled data, via models like mBERT and XLM-R (100+ languages from raw text).
• Disparities persist: the tokenizer tax (low-resource languages split into many more subword tokens) and the curse of multilinguality (fixed capacity dilutes per-language quality).
• LLMs are multilingual by default but English-skewed; the shared-space idea explains both the reach and the inequity.

Quick check

Next

The shared space builds on word embeddings and subword units; the multilingual encoders are BERT-family models (mBERT, XLM-R). Modern LLMs inherit both the reach and the English skew.