Learning to rank

At 3:00 a.m., someone searches your shop for “running shoes under $100.” The first result is a $240 trail shoe. The second is a $79 road shoe with the right size in stock. Your model gave the expensive shoe a relevance score of 0.81 and the useful shoe 0.79.

The model was not asked to classify either result as “relevant” or “irrelevant.” It was asked to put the useful things first.

That distinction matters. Search, recommendations, news feeds, job boards, and sponsored listings all produce an ordered list. Users usually inspect only the first few items. A model that predicts each document accurately in isolation can still make a terrible list.

Learning to rank means training a model whose output is useful as an ordering. Better items should receive higher scores than worse items for the same query.

Why pointwise regression can lose the plot

A query is the user’s request. A document is any item that could appear for it. A relevance label measures how well that document answers the query; an editorial scale might run from 0, “not relevant,” to 4, “excellent.”

A pointwise model sees one query-document pair at a time and predicts its label. With regression, the usual loss is mean squared error: the average squared gap between predictions and observed relevance.

That is sensible when the number itself matters, but less so when only the order matters.

Consider two documents for one query:

Document	True relevance	Predicted score
A	3	2.49
B	2	2.51

The model puts B above A, which is wrong. Its mean squared error is still only:

((3 - 2.49)² + (2 - 2.51)²) / 2 = 0.2601

An RMSE of about 0.51 relevance points on a zero-to-four scale does not look catastrophic. Yet the first result is the less useful one.

The cause is structural. Squared error asks, “How close was each prediction to its label?” Ranking asks, “Was A’s score greater than B’s?” A pointwise loss does not apply a special penalty when two neighboring documents are swapped. Across a large candidate set, one reversed top-two pair can barely change average loss while changing the user’s first click.

Predicting the average relevance for every document can even be a respectable noisy-label baseline. It still creates no useful order. Ranking scores are also invariant to any increasing transformation: 0.8 and 0.6 produce the same order as 80 and 60, while pointwise regression cares about their scale.

Pointwise learning remains useful as a simple baseline, or when the predicted value itself matters. It is just not the natural objective for “put the best items first.”

Ranking metrics: reward the order users actually see

Ranking metrics evaluate a whole list per query, then average across queries. They do not compare documents from unrelated queries.

MRR

Mean reciprocal rank, or MRR, measures where the first relevant result appears. For one query, the reciprocal rank is 1 / position: a relevant result at position 1 scores 1, and one at position 4 scores 0.25.

MRR fits fact-finding searches such as “reset my password.” It ignores every relevant result after the first, so it is a poor fit for a shopping page where several results may be useful.

MAP

Mean average precision, or MAP, rewards retrieving several binary-relevant documents early.

For a query, let R be the total number of relevant documents in the judged collection, including relevant documents the system failed to retrieve. Let rel_i be 1 when the document at rank i is relevant and 0 otherwise:

AP@k = (1 / R) * sum(i=1..k, precision@i * rel_i)

Only relevant ranks contribute, but the denominator remains R. Thus a result list cannot hide relevant documents it missed. For R = 0, teams commonly assign AP 0 or omit the query; use one policy consistently.

MAP works when relevance is clearly binary and several relevant results matter. It cannot naturally express that an excellent result is much better than a merely acceptable one.

NDCG

Normalized discounted cumulative gain, or NDCG, handles graded relevance and emphasizes early positions.

First calculate:

DCG@k = sum((2^relevance - 1) / log2(position + 1))

The gain term makes a relevance-3 result worth more than three relevance-1 results. The logarithm discounts lower positions. Divide by the ideal DCG, IDCG, obtained by sorting all judged labels from best to worst:

NDCG@k = DCG@k / IDCG@k

Suppose the ideal labels are 3, 2, 0, but the model returns 2, 0, 3. At position 3, the relevance-3 item contributes 7 / log2(4) = 3.5; the model’s total is:

DCG@3 = 3 + 0 + 3.5 = 6.5

The ideal total is:

IDCG@3 = 7 / log2(2) + 3 / log2(3) + 0 ≈ 8.893

Therefore:

NDCG@3 = 6.5 / 8.893 ≈ 0.731

import math


def dcg_at_k(relevances, k):
    return sum(
        ((2 ** relevance) - 1) / math.log2(position + 2)
        for position, relevance in enumerate(relevances[:k])
    )


ranking = [2, 0, 3]
all_relevances = [3, 2, 0]
k = 3
ideal = sorted(all_relevances, reverse=True)[:k]

dcg = dcg_at_k(ranking, k)
idcg = dcg_at_k(ideal, k)

print(f"DCG@3 = {dcg:.3f}")
print(f"IDCG@3 = {idcg:.3f}")
print(f"NDCG@3 = {dcg / idcg:.3f}")

It prints:

DCG@3 = 6.500
IDCG@3 = 8.893
NDCG@3 = 0.731

Build the ideal ranking from the complete set of judged relevances, not only returned documents. Otherwise, missing relevant documents disappear from the denominator. If every label is 0, IDCG is 0; skip such queries or assign NDCG 0, consistently.

For most search and recommendation pages, NDCG at the visible cutoff—NDCG@3, @10, or @20—is the most expressive of these metrics. The cutoff should match the product surface: NDCG@100 can improve while the first screen gets worse.

See Picking the right metric for the broader discipline of matching a metric to a decision.

Three ways to train the order

Pointwise: predict each label

Pointwise training gives every query-document pair its own target, using squared error for graded labels or logistic loss for binary relevance. It is cheap and easy to debug, but it ignores relationships among documents in the same result list.

Pairwise: learn which document should win

Pairwise learning trains on preferences between two documents from the same query. If A has a higher relevance label than B, the example says A should rank above B.

RankNet turns the score difference into a probability:

P(A above B) = sigmoid(s_A - s_B)

The target is 1 when A should win, and the model minimizes binary cross-entropy. For one preferred pair, the gradient for A is:

sigmoid(s_A - s_B) - 1

B receives the opposite gradient. If A is already far above B, the update is small. If B is incorrectly above A, the update is large: increase A’s score and decrease B’s.

RankNet therefore asks whether the difference s_A - s_B has the right sign and confidence, rather than whether A’s score equals a particular label.

Pair construction can be expensive: 1,000 documents produce nearly half a million unordered pairs. Systems sample pairs, focus on confusing pairs, or use documents that appeared together. Pairwise loss also does not know that position 1 may matter more than position 20.

Listwise and metric-aware training

A listwise objective considers all documents for a query as one list. Some methods model permutations; others define a loss over the whole score vector.

LambdaRank takes a practical shortcut. It starts with a RankNet-style pairwise gradient, then weights each pair by how much swapping it would change the target metric:

lambda_A,B = metric_change × pairwise_error

The resulting update encodes:

1. Direction: A should move up and B down.
2. Confidence: an incorrectly ordered or uncertain pair needs more correction.
3. Impact: a swap near the top matters more than one near the bottom.

For the same relevance-3 versus relevance-1 swap, the NDCG change is larger between positions 1 and 2 than between positions 50 and 51 because of position discount. LambdaRank therefore sends a larger update to the first pair.

LambdaMART fits these lambda pseudo-gradients with MART-style gradient-boosted regression trees. It is effective for structured tabular features such as text matching, freshness, price, popularity, user-item history, device, geography, and availability.

LambdaRank and LambdaMART are often called listwise because their gradients depend on the whole ranked list and can target NDCG. Strictly, their updates are assembled from pairs: they are metric-aware pairwise methods, not full-permutation probability models.

Method	Training signal	Main use	Main blind spot
Pointwise	Each item’s label	Simple baseline or reliable value labels	Does not directly optimize pair order
RankNet	Preferred within-query pairs	Straightforward pairwise supervision	Usually ignores position impact
LambdaRank	Pair gradients weighted by metric change	Top positions and NDCG-like objectives matter	More complex grouping and debugging
LambdaMART	Lambda updates fitted by boosted trees	Strong tabular production ranker	Needs good retrieval and semantic features

Rankers need query groups. If rows are shuffled without preserving which documents belong to each query, pairwise and listwise training has lost its structure. Split by time, user, or query family where appropriate; random splits can leak information from one search session into both train and test. Data leakage is particularly easy to create in ranking logs.

Clicks are observations, not relevance labels

A click log records what a user did after seeing a ranked page, not whether the item was relevant. This is position bias: the first result is examined more often, and a user may click it because it is useful, attractive, or simply visible. A relevant result lower down can receive no click.

The log also comes from the old ranker. If that system rarely places a document in the top five, it collects little evidence about how users would respond to it there. Training directly on those clicks can preserve the old ordering and its mistakes. Snippet wording, price, thumbnails, device, familiarity, and query intent add further confounding.

Useful countermeasures include randomized result swaps, interleaving two rankers, propensity weighting based on examination probability, and editorial judgments. Randomization creates evidence at positions the old ranker would not have used, but costs some user experience and operational risk. Keep experiments bounded and monitor guardrails.

The production pattern: retrieve, then rerank

A search system usually cannot run an expensive ranker over every document. For a catalog of 10 million products, an inverted index, vector index, or both might retrieve 1,000 candidates. A second-stage ranker scores those candidates with richer features, then serves the best 10.

Retrieval is optimized for recall@1,000: the fraction of useful items present in the candidate set. Reranking spends more computation only where it can affect the page.

If the useful running shoe never enters the 1,000 candidates, the reranker cannot rescue it. A high-recall retriever can also return a noisy set that overwhelms the reranker’s budget. Increasing candidates may improve recall while worsening latency, cost, and timeouts, so measure the trade-off on real traffic and hardware.

The classic failure: offline wins, users lose

Offline evaluation uses held-out lists and labels. Online evaluation measures real behavior: successful task completion, conversion, retention, complaints, latency, and sometimes clicks or satisfaction. They can disagree because they measure different worlds.

A model might raise NDCG@10 on click-derived data while hurting users because it learned position bias, rewards topical similarity instead of task completion, produces repetitive results, or adds latency. The test set may also reflect the old exposure policy, while the new model changes which documents receive exposure.

When offline relevance rises but online outcomes fall, inspect the label source, exposure policy, and exact online event before tuning model parameters.

There is no universally best ranking objective. Start with pointwise learning for a transparent baseline, RankNet when pair preferences are clean supervision, and LambdaRank or LambdaMART when top positions and the page metric justify grouped training.

The honest limitation is that learning to rank cannot manufacture relevance labels. It optimizes the feedback supplied to it, including biased clicks, stale judgments, popularity loops, and accidental business incentives. A flawless NDCG implementation can still optimize the wrong idea of “good.”

What to remember

• Ranking is about relative order within a query, not accurate predictions in isolation.
• Pointwise loss can be low while a crucial pair is reversed; pairwise objectives train the ordering directly.
• RankNet supplies direction and confidence. LambdaRank adds metric impact. LambdaMART fits those updates with boosted trees.
• MRR cares about the first relevant result, MAP about several binary-relevant results, and NDCG about graded relevance with position discounts.
• Clicks reflect behavior under exposure. Retrieval recall, latency, and online outcomes matter as much as offline rank metrics.

Quick check

0/3

Q1

Mean squared error evaluates each document independently and does not directly penalize pair reversalsNDCG can only be used for binary labelsPairwise losses remove the need for relevance labelsThe model must be using too many trees

Check answer

Q2

Documents near position 1 always have larger feature vectorsThe metric change from swapping them is larger because NDCG discounts lower positionsLambdaRank ignores the target metric during trainingThe first position is excluded from the training list

Check answer

Q3

Whether the model's tree leaves are all the same sizeWhether the offline labels and NDCG objective match watch-quality, freshness, and diversity in the live feedWhether MRR is always greater than MAPWhether retrieval should be removed so the reranker sees every video

Check answer